nim-serde/serde/json

nim-serde JSON

The JSON module in nim-serde provides serialization and deserialization for Nim values, offering an improved alternative to the standard std/json library. Unlike the standard library, nim-serde JSON implements a flexible system of serialization/deserialization modes that give developers precise control over how Nim objects are converted to and from JSON.

Table of Contents

• nim-serde JSON
  • Table of Contents
  • Serde Modes
    • Modes Overview
    • Default Modes
    • Field Options
  • Serialization API
    • Basic Serialization with % operator
    • Object Serialization
    • Inlining JSON Directly in Code with %*
    • Converting to JSON String with toJson
    • Serialization Modes
    • Field Customization for Serialization
  • Custom Type Serialization
  • Deserialization API
    • Basic Deserialization with fromJson
    • Error Handling
    • Parsing JSON with JsonNode.parse
    • Deserialization Modes
    • Field Customization for Deserialization
  • Using as a Drop-in Replacement for std/json
  • Implementation Details

Serde Modes

This implementation supports three different modes to control de/serialization:

OptIn
OptOut
Strict

Modes can be set in the {.serialize.} and/or {.deserialize.} pragmas on type definitions. Each mode has a different meaning depending on if the type is being serialized or deserialized. Modes can be set by setting mode in the serialize or deserialize pragma annotation, eg:

type MyType {.serialize(mode=Strict).} = object
  field1: bool
  field2: bool

Modes Overview

Mode	Serialize	Deserialize
OptOut	All object fields will be serialized, except fields marked with {.serialize(ignore=true).}.	All JSON keys will be deserialized, except fields marked with {.deserialize(ignore=true).}. No error if extra JSON fields exist.
OptIn	Only fields marked with {.serialize.} will be serialized. Fields marked with {.serialize(ignore=true).} will not be serialized.	Only fields marked with {.deserialize.} will be deserialized. Fields marked with {.deserialize(ignore=true).} will not be deserialized. A SerdeError is raised if the field is missing in JSON.
Strict	All object fields will be serialized, regardless if the field is marked with {.serialize(ignore=true).}.	Object fields and JSON fields must match exactly, otherwise a SerdeError is raised.

Default Modes

Types can be serialized and deserialized even without explicit annotations, using default modes. Without any pragmas, types are serialized in OptIn mode and deserialized in OptOut mode. When types have pragmas but no specific mode is set, OptOut mode is used for both serialization and deserialization.

Context	Serialize	Deserialize
Default (no pragma)	OptIn	OptOut
Default (pragma, but no mode)	OptOut	OptOut

# Type is not annotated
# If you don't annotate the type, serde assumes OptIn by default for serialization, and OptOut for 
# deserialization. This means your types will be serialized to an empty string, which is probably not what you want:
type MyObj1 = object
  field1: bool
  field2: bool

# If you annotate your type but do not specify the mode, serde will default to OptOut for
# both serialize and de-serialize, meaning all fields get serialized/de-serialized by default:
# A default mode of OptOut is assumed for both serialize and deserialize.
type MyObj2 {.serialize, deserialize.} = object
  field1: bool
  field2: bool

Field Options

Individual fields can be customized using the {.serialize.} and {.deserialize.} pragmas with additional options that control how each field is processed during serialization and deserialization

	serialize	deserialize
key	aliases the field name in json	deserializes the field if json contains key
ignore	OptOut: field not serialized OptIn: field not serialized Strict: field serialized	OptOut: field not deserialized OptIn: field not deserialized Strict: field deserialized

Example with field options:

import pkg/serde/json

type
  Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
    id {.serialize(ignore=true), deserialize(key="personid").}: int
    name: string
    birthYear: int
    address: string
    phone: string

let person = Person(
              name: "Lloyd Christmas",
              birthYear: 1970,
              address: "123 Sesame Street, Providence, Rhode Island  12345",
              phone: "555-905-justgivemethedamnnumber!⛽️🔥")

let createRequest = """{
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert person.toJson(pretty=true) == createRequest

let createResponse = """{
  "personid": 1,
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert !Person.fromJson(createResponse) == Person(id: 1)

More examples can be found in Serialization Modes and Deserialization Modes.

Serialization API

The nim-serde JSON serialization API provides several ways to convert Nim values to JSON.

Basic Serialization with % operator

The % operator converts Nim values to JsonNode objects, which can then be converted to JSON strings:

import pkg/serde/json

# Basic types
assert %42 == newJInt(42)
assert %"hello" == newJString("hello")
assert %true == newJBool(true)

# Arrays and sequences
let arr = newJArray()
arr.add(newJInt(1))
arr.add(newJInt(2))
arr.add(newJInt(3))
assert $(%[1, 2, 3]) == $arr

Object Serialization

Objects can be serialized using the % operator, which automatically handles field serialization based on the object's configuration:

import pkg/serde/json

type Person = object
  name {.serialize.}: string
  age {.serialize.}: int
  address: string  # Not serialized by default in OptIn mode

let person = Person(name: "John", age: 30, address: "123 Main St")
let jsonNode = %person
assert jsonNode.kind == JObject
assert jsonNode.len == 2
assert jsonNode["name"].getStr == "John"
assert jsonNode["age"].getInt == 30
assert "address" notin jsonNode

Inlining JSON Directly in Code with %*

The %* macro provides a more convenient way to create JSON objects:

import pkg/serde/json

let
  name = "John"
  age = 30
  jsonObj = %*{
  "name": name,
  "age": age,
  "hobbies": ["reading", "coding"],
  "address": {
    "street": "123 Main St",
    "city": "Anytown"
  }
}

assert jsonObj.kind == JObject
assert jsonObj["name"].getStr == name
assert jsonObj["age"].getInt == age
assert jsonObj["hobbies"].kind == JArray
assert jsonObj["hobbies"][0].getStr == "reading"
assert jsonObj["address"]["street"].getStr == "123 Main St"

Converting to JSON String with toJson

The toJson function converts any serializable value directly to a JSON string:

import pkg/serde/json

type Person = object
  name {.serialize.}: string
  age {.serialize.}: int

let person = Person(name: "John", age: 30)
assert person.toJson == """{"name":"John","age":30}"""

Serialization Modes

nim-serde offers three modes to control which fields are serialized:

import pkg/serde/json

# OptIn mode (default): Only fields with {.serialize.} are included
type Person1 = object
  name {.serialize.}: string
  age {.serialize.}: int
  address: string  # Not serialized

assert Person1(name: "John", age: 30, address: "123 Main St").toJson == """{"name":"John","age":30}"""

# OptOut mode: All fields are included except those marked to ignore
type Person2 {.serialize.} = object
  name: string
  age: int
  ssn {.serialize(ignore=true).}: string  # Not serialized

assert Person2(name: "John", age: 30, ssn: "123-45-6789").toJson == """{"name":"John","age":30}"""

# Strict mode: All fields are included, and an error is raised if any fields are missing
type Person3 {.serialize(mode=Strict).} = object
  name: string
  age: int

assert Person3(name: "John", age: 30).toJson == """{"name":"John","age":30}"""

Field Customization for Serialization

Fields can be customized with various options:

import pkg/serde/json

# Field customization for serialization
type Person {.serialize(mode = OptOut).} = object
  firstName {.serialize(key = "first_name").}: string
  lastName {.serialize(key = "last_name").}: string
  age: int  # Will be included because we're using OptOut mode
  ssn {.serialize(ignore = true).}: string  # Sensitive data not serialized

let person = Person(
  firstName: "John",
  lastName: "Doe",
  age: 30,
  ssn: "123-45-6789"
)

let jsonNode = %person
assert jsonNode.kind == JObject
assert jsonNode["first_name"].getStr == "John"
assert jsonNode["last_name"].getStr == "Doe"
assert jsonNode["age"].getInt == 30
assert "ssn" notin jsonNode

# Convert to JSON string
let jsonStr = toJson(person)
assert jsonStr == """{"first_name":"John","last_name":"Doe","age":30}"""

Custom Type Serialization

You can extend nim-serde to support custom types by defining your own % operator overloads and fromJson procs:

import pkg/serde/json
import pkg/serde/utils/errors
import pkg/questionable/results
import std/strutils

# Define a custom type
type
  UserId = distinct int

# Custom serialization for UserId
proc `%`*(id: UserId): JsonNode =
  %("user-" & $int(id))

# Custom deserialization for UserId
proc fromJson*(_: type UserId, json: JsonNode): ?!UserId =
  if json.kind != JString:
    return failure(newSerdeError("Expected string for UserId, got " & $json.kind))
  
  let str = json.getStr()
  if str.startsWith("user-"):
    let idStr = str[5..^1]
    try:
      let id = parseInt(idStr)
      success(UserId(id))
    except ValueError:
      failure(newSerdeError("Invalid UserId format: " & str))
  else:
    failure(newSerdeError("UserId must start with 'user-' prefix"))

# Test serialization
let userId = UserId(42)
let jsonNode = %userId
assert jsonNode.kind == JString
assert jsonNode.getStr() == "user-42"

# Test deserialization
let jsonStr = "\"user-42\""
let parsedJson = !JsonNode.parse(jsonStr)
let result = UserId.fromJson(parsedJson)
assert result.isSuccess
assert int(!result) == 42

# Test in object context
type User {.serialize(mode = OptOut).} = object
  id: UserId
  name: string

let user = User(id: UserId(123), name: "John")
let userJson = %user
assert userJson.kind == JObject
assert userJson["id"].getStr() == "user-123"
assert userJson["name"].getStr() == "John"

# Test deserialization of object with custom type
let userJsonStr = """{"id":"user-123","name":"John"}"""
let userResult = User.fromJson(userJsonStr)
assert userResult.isSuccess
assert int((!userResult).id) == 123
assert (!userResult).name == "John"

Deserialization API

nim-serde provides a type-safe way to convert JSON data back into Nim types.

Basic Deserialization with fromJson

The fromJson function converts JSON strings or JsonNode objects to Nim types:

import pkg/serde/json
import pkg/questionable/results

type Person = object
  name: string
  age: int

let jsonStr = """{"name":"John","age":30}"""
let result = Person.fromJson(jsonStr)

# Using the ! operator from questionable to extract the value
assert !result == Person(name: "John", age: 30)

Error Handling

Deserialization returns a Result type from the questionable library, allowing for safe error handling:

import pkg/serde/json
import pkg/questionable/results

type Person = object
  name: string
  age: int

let invalidJson = """{"name":"John","age":"thirty"}"""
let errorResult = Person.fromJson(invalidJson)
assert errorResult.isFailure
assert errorResult.error of UnexpectedKindError

Parsing JSON with JsonNode.parse

To parse JSON string into a JsonNode tree instead of a deserializing to a concrete type, use JsonNode.parse:

import pkg/serde/json
import pkg/questionable/results

let jsonStr = """{"name":"John","age":30,"hobbies":["reading","coding"]}"""
let parseResult = JsonNode.parse(jsonStr)
assert parseResult.isSuccess
let jsonNode = !parseResult
assert jsonNode["name"].getStr == "John"
assert jsonNode["age"].getInt == 30
assert jsonNode["hobbies"].kind == JArray
assert jsonNode["hobbies"][0].getStr == "reading"
assert jsonNode["hobbies"][1].getStr == "coding"

Deserialization Modes

nim-serde offers three modes to control how JSON is deserialized:

import pkg/serde/json
import pkg/questionable/results

# OptOut mode (default for deserialization)
type PersonOptOut = object
  name: string
  age: int

let jsonOptOut = """{"name":"John","age":30,"address":"123 Main St"}"""
let resultOptOut = PersonOptOut.fromJson(jsonOptOut)
assert resultOptOut.isSuccess
assert !resultOptOut == PersonOptOut(name: "John", age: 30)

# OptIn mode
type PersonOptIn {.deserialize(mode = OptIn).} = object
  name {.deserialize.}: string
  age {.deserialize.}: int
  address: string  # Not deserialized by default in OptIn mode

let jsonOptIn = """{"name":"John","age":30,"address":"123 Main St"}"""
let resultOptIn = PersonOptIn.fromJson(jsonOptIn)
assert resultOptIn.isSuccess
assert (!resultOptIn).name == "John"
assert (!resultOptIn).age == 30
assert (!resultOptIn).address == ""  # address is not deserialized

# Strict mode
type PersonStrict {.deserialize(mode = Strict).} = object
  name: string
  age: int

let jsonStrict = """{"name":"John","age":30}"""
let resultStrict = PersonStrict.fromJson(jsonStrict)
assert resultStrict.isSuccess
assert !resultStrict == PersonStrict(name: "John", age: 30)

# Strict mode with extra field (should fail)
let jsonStrictExtra = """{"name":"John","age":30,"address":"123 Main St"}"""
let resultStrictExtra = PersonStrict.fromJson(jsonStrictExtra)
assert resultStrictExtra.isFailure

Field Customization for Deserialization

Fields can be customized with various options for deserialization:

import pkg/serde/json
import pkg/questionable/results

type User = object
  firstName {.deserialize(key = "first_name").}: string
  lastName {.deserialize(key = "last_name").}: string
  age: int
  internalId {.deserialize(ignore = true).}: int

let userJsonStr = """{"first_name":"Jane","last_name":"Smith","age":25,"role":"admin"}"""
let result = User.fromJson(userJsonStr)
assert result.isSuccess
assert (!result).firstName == "Jane"
assert (!result).lastName == "Smith"
assert (!result).age == 25
assert (!result).internalId == 0  # Default value, not deserialized

Using as a Drop-in Replacement for std/json

nim-serde can be used as a drop-in replacement for the standard library's json module with improved exception handling.

# Instead of:
# import std/json
import pkg/serde/json
import pkg/questionable/results

# Using nim-serde's JSON API which is compatible with std/json
let jsonNode = %* {
  "name": "John",
  "age": 30,
  "isActive": true,
  "hobbies": ["reading", "swimming"]
}

# Accessing JSON fields using the same API as std/json
assert jsonNode.kind == JObject
assert jsonNode["name"].getStr == "John"
assert jsonNode["age"].getInt == 30
assert jsonNode["isActive"].getBool == true
assert jsonNode["hobbies"].kind == JArray
assert jsonNode["hobbies"][0].getStr == "reading"
assert jsonNode["hobbies"][1].getStr == "swimming"

# Converting JSON to string
let jsonStr = $jsonNode

# Parsing JSON from string with better error handling
let parsedResult = JsonNode.parse(jsonStr)
assert parsedResult.isSuccess
let parsedNode = !parsedResult
assert parsedNode.kind == JObject
assert parsedNode["name"].getStr == "John"

# Pretty printing
let prettyJson = pretty(jsonNode)

Implementation Details

The JSON serialization in nim-serde is based on the % operator pattern:

1. The % operator converts values to JsonNode objects
2. Various overloads handle different types (primitives, objects, collections)
3. The toJson function converts the JsonNode to a string

This approach makes it easy to extend with custom types by defining your own % operator overloads.

For deserialization, the library uses compile-time reflection to map JSON fields to object fields, respecting the configuration provided by pragmas.