mirror of
https://github.com/logos-storage/nim-serde.git
synced 2026-01-08 00:23:07 +00:00
534 lines
17 KiB
Markdown
534 lines
17 KiB
Markdown
# nim-serde JSON
|
|
|
|
The JSON module in nim-serde provides serialization and deserialization for Nim values, offering an improved alternative to the standard `std/json` library. Unlike the standard library, nim-serde JSON implements a flexible system of serialization/deserialization modes that give developers precise control over how Nim objects are converted to and from JSON.
|
|
|
|
## Table of Contents
|
|
- [nim-serde JSON](#nim-serde-json)
|
|
- [Table of Contents](#table-of-contents)
|
|
- [Serde Modes](#serde-modes)
|
|
- [Modes Overview](#modes-overview)
|
|
- [Default Modes](#default-modes)
|
|
- [Field Options](#field-options)
|
|
- [Serialization API](#serialization-api)
|
|
- [Basic Serialization with `%` operator](#basic-serialization-with--operator)
|
|
- [Object Serialization](#object-serialization)
|
|
- [Inlining JSON Directly in Code with `%*`](#inlining-json-directly-in-code-with-)
|
|
- [Converting to JSON String with `toJson`](#converting-to-json-string-with-tojson)
|
|
- [Serialization Modes](#serialization-modes)
|
|
- [Field Customization for Serialization](#field-customization-for-serialization)
|
|
- [Custom Type Serialization](#custom-type-serialization)
|
|
- [Deserialization API](#deserialization-api)
|
|
- [Basic Deserialization with `fromJson`](#basic-deserialization-with-fromjson)
|
|
- [Error Handling](#error-handling)
|
|
- [Parsing JSON with `JsonNode.parse`](#parsing-json-with-jsonnodeparse)
|
|
- [Deserialization Modes](#deserialization-modes)
|
|
- [Field Customization for Deserialization](#field-customization-for-deserialization)
|
|
- [Using as a Drop-in Replacement for std/json](#using-as-a-drop-in-replacement-for-stdjson)
|
|
- [Implementation Details](#implementation-details)
|
|
|
|
|
|
## Serde Modes
|
|
This implementation supports three different modes to control de/serialization:
|
|
|
|
```nim
|
|
OptIn
|
|
OptOut
|
|
Strict
|
|
```
|
|
|
|
Modes can be set in the `{.serialize.}` and/or `{.deserialize.}` pragmas on type
|
|
definitions. Each mode has a different meaning depending on if the type is being
|
|
serialized or deserialized. Modes can be set by setting `mode` in the `serialize` or
|
|
`deserialize` pragma annotation, eg:
|
|
|
|
```nim
|
|
type MyType {.serialize(mode=Strict).} = object
|
|
field1: bool
|
|
field2: bool
|
|
```
|
|
|
|
### Modes Overview
|
|
|
|
| Mode | Serialize | Deserialize |
|
|
|:-----|:----------|:------------|
|
|
| `OptOut` | All object fields will be serialized, except fields marked with `{.serialize(ignore=true).}`. | All JSON keys will be deserialized, except fields marked with `{.deserialize(ignore=true).}`. No error if extra JSON fields exist. |
|
|
| `OptIn` | Only fields marked with `{.serialize.}` will be serialized. Fields marked with `{.serialize(ignore=true).}` will not be serialized. | Only fields marked with `{.deserialize.}` will be deserialized. Fields marked with `{.deserialize(ignore=true).}` will not be deserialized. A `SerdeError` is raised if the field is missing in JSON. |
|
|
| `Strict` | All object fields will be serialized, regardless if the field is marked with `{.serialize(ignore=true).}`. | Object fields and JSON fields must match exactly, otherwise a `SerdeError` is raised. |
|
|
|
|
### Default Modes
|
|
|
|
Types can be serialized and deserialized even without explicit annotations, using default modes. Without any pragmas, types are serialized in OptIn mode and deserialized in OptOut mode. When types have pragmas but no specific mode is set, OptOut mode is used for both serialization and deserialization.
|
|
|
|
|
|
| Context | Serialize | Deserialize |
|
|
|:--------|:----------|:------------|
|
|
| Default (no pragma) | `OptIn` | `OptOut` |
|
|
| Default (pragma, but no mode) | `OptOut` | `OptOut` |
|
|
|
|
```nim
|
|
# Type is not annotated
|
|
# If you don't annotate the type, serde assumes OptIn by default for serialization, and OptOut for
|
|
# deserialization. This means your types will be serialized to an empty string, which is probably not what you want:
|
|
type MyObj1 = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
# If you annotate your type but do not specify the mode, serde will default to OptOut for
|
|
# both serialize and de-serialize, meaning all fields get serialized/de-serialized by default:
|
|
# A default mode of OptOut is assumed for both serialize and deserialize.
|
|
type MyObj2 {.serialize, deserialize.} = object
|
|
field1: bool
|
|
field2: bool
|
|
```
|
|
|
|
### Field Options
|
|
|
|
Individual fields can be customized using the `{.serialize.}` and `{.deserialize.}` pragmas with additional options that control how each field is processed during serialization and deserialization
|
|
|
|
|
|
| | serialize | deserialize |
|
|
|:---------|:-----------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------|
|
|
| `key` | aliases the field name in json | deserializes the field if json contains `key` |
|
|
| `ignore` | <li>**OptOut:** field not serialized</li><li>**OptIn:** field not serialized</li><li>**Strict:** field serialized</li> | <li>**OptOut:** field not deserialized</li><li>**OptIn:** field not deserialized</li><li>**Strict:** field deserialized</li> |
|
|
|
|
|
|
Example with field options:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type
|
|
Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
|
|
id {.serialize(ignore=true), deserialize(key="personid").}: int
|
|
name: string
|
|
birthYear: int
|
|
address: string
|
|
phone: string
|
|
|
|
let person = Person(
|
|
name: "Lloyd Christmas",
|
|
birthYear: 1970,
|
|
address: "123 Sesame Street, Providence, Rhode Island 12345",
|
|
phone: "555-905-justgivemethedamnnumber!⛽️🔥")
|
|
|
|
let createRequest = """{
|
|
"name": "Lloyd Christmas",
|
|
"birthYear": 1970,
|
|
"address": "123 Sesame Street, Providence, Rhode Island 12345",
|
|
"phone": "555-905-justgivemethedamnnumber!⛽️🔥"
|
|
}"""
|
|
assert person.toJson(pretty=true) == createRequest
|
|
|
|
let createResponse = """{
|
|
"personid": 1,
|
|
"name": "Lloyd Christmas",
|
|
"birthYear": 1970,
|
|
"address": "123 Sesame Street, Providence, Rhode Island 12345",
|
|
"phone": "555-905-justgivemethedamnnumber!⛽️🔥"
|
|
}"""
|
|
assert !Person.fromJson(createResponse) == Person(id: 1)
|
|
```
|
|
|
|
More examples can be found in [Serialization Modes](#serialization-modes) and [Deserialization Modes](#deserialization-modes).
|
|
|
|
## Serialization API
|
|
|
|
The nim-serde JSON serialization API provides several ways to convert Nim values to JSON.
|
|
|
|
### Basic Serialization with `%` operator
|
|
|
|
The `%` operator converts Nim values to `JsonNode` objects, which can then be converted to JSON strings:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
# Basic types
|
|
assert %42 == newJInt(42)
|
|
assert %"hello" == newJString("hello")
|
|
assert %true == newJBool(true)
|
|
|
|
# Arrays and sequences
|
|
let arr = newJArray()
|
|
arr.add(newJInt(1))
|
|
arr.add(newJInt(2))
|
|
arr.add(newJInt(3))
|
|
assert $(%[1, 2, 3]) == $arr
|
|
```
|
|
|
|
### Object Serialization
|
|
|
|
Objects can be serialized using the `%` operator, which automatically handles field serialization based on the object's configuration:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type Person = object
|
|
name {.serialize.}: string
|
|
age {.serialize.}: int
|
|
address: string # Not serialized by default in OptIn mode
|
|
|
|
let person = Person(name: "John", age: 30, address: "123 Main St")
|
|
let jsonNode = %person
|
|
assert jsonNode.kind == JObject
|
|
assert jsonNode.len == 2
|
|
assert jsonNode["name"].getStr == "John"
|
|
assert jsonNode["age"].getInt == 30
|
|
assert "address" notin jsonNode
|
|
```
|
|
|
|
### Inlining JSON Directly in Code with `%*`
|
|
|
|
The `%*` macro provides a more convenient way to create JSON objects:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
let
|
|
name = "John"
|
|
age = 30
|
|
jsonObj = %*{
|
|
"name": name,
|
|
"age": age,
|
|
"hobbies": ["reading", "coding"],
|
|
"address": {
|
|
"street": "123 Main St",
|
|
"city": "Anytown"
|
|
}
|
|
}
|
|
|
|
assert jsonObj.kind == JObject
|
|
assert jsonObj["name"].getStr == name
|
|
assert jsonObj["age"].getInt == age
|
|
assert jsonObj["hobbies"].kind == JArray
|
|
assert jsonObj["hobbies"][0].getStr == "reading"
|
|
assert jsonObj["address"]["street"].getStr == "123 Main St"
|
|
```
|
|
|
|
### Converting to JSON String with `toJson`
|
|
|
|
The `toJson` function converts any serializable value directly to a JSON string:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type Person = object
|
|
name {.serialize.}: string
|
|
age {.serialize.}: int
|
|
|
|
let person = Person(name: "John", age: 30)
|
|
assert person.toJson == """{"name":"John","age":30}"""
|
|
```
|
|
|
|
### Serialization Modes
|
|
|
|
nim-serde offers three modes to control which fields are serialized:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
# OptIn mode (default): Only fields with {.serialize.} are included
|
|
type Person1 = object
|
|
name {.serialize.}: string
|
|
age {.serialize.}: int
|
|
address: string # Not serialized
|
|
|
|
assert Person1(name: "John", age: 30, address: "123 Main St").toJson == """{"name":"John","age":30}"""
|
|
|
|
# OptOut mode: All fields are included except those marked to ignore
|
|
type Person2 {.serialize.} = object
|
|
name: string
|
|
age: int
|
|
ssn {.serialize(ignore=true).}: string # Not serialized
|
|
|
|
assert Person2(name: "John", age: 30, ssn: "123-45-6789").toJson == """{"name":"John","age":30}"""
|
|
|
|
# Strict mode: All fields are included, and an error is raised if any fields are missing
|
|
type Person3 {.serialize(mode=Strict).} = object
|
|
name: string
|
|
age: int
|
|
|
|
assert Person3(name: "John", age: 30).toJson == """{"name":"John","age":30}"""
|
|
```
|
|
|
|
### Field Customization for Serialization
|
|
|
|
Fields can be customized with various options:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
# Field customization for serialization
|
|
type Person {.serialize(mode = OptOut).} = object
|
|
firstName {.serialize(key = "first_name").}: string
|
|
lastName {.serialize(key = "last_name").}: string
|
|
age: int # Will be included because we're using OptOut mode
|
|
ssn {.serialize(ignore = true).}: string # Sensitive data not serialized
|
|
|
|
let person = Person(
|
|
firstName: "John",
|
|
lastName: "Doe",
|
|
age: 30,
|
|
ssn: "123-45-6789"
|
|
)
|
|
|
|
let jsonNode = %person
|
|
assert jsonNode.kind == JObject
|
|
assert jsonNode["first_name"].getStr == "John"
|
|
assert jsonNode["last_name"].getStr == "Doe"
|
|
assert jsonNode["age"].getInt == 30
|
|
assert "ssn" notin jsonNode
|
|
|
|
# Convert to JSON string
|
|
let jsonStr = toJson(person)
|
|
assert jsonStr == """{"first_name":"John","last_name":"Doe","age":30}"""
|
|
```
|
|
|
|
## Custom Type Serialization
|
|
|
|
You can extend nim-serde to support custom types by defining your own `%` operator overloads and `fromJson` procs:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
import pkg/serde/utils/errors
|
|
import pkg/questionable/results
|
|
import std/strutils
|
|
|
|
# Define a custom type
|
|
type
|
|
UserId = distinct int
|
|
|
|
# Custom serialization for UserId
|
|
proc `%`*(id: UserId): JsonNode =
|
|
%("user-" & $int(id))
|
|
|
|
# Custom deserialization for UserId
|
|
proc fromJson*(_: type UserId, json: JsonNode): ?!UserId =
|
|
if json.kind != JString:
|
|
return failure(newSerdeError("Expected string for UserId, got " & $json.kind))
|
|
|
|
let str = json.getStr()
|
|
if str.startsWith("user-"):
|
|
let idStr = str[5..^1]
|
|
try:
|
|
let id = parseInt(idStr)
|
|
success(UserId(id))
|
|
except ValueError:
|
|
failure(newSerdeError("Invalid UserId format: " & str))
|
|
else:
|
|
failure(newSerdeError("UserId must start with 'user-' prefix"))
|
|
|
|
# Test serialization
|
|
let userId = UserId(42)
|
|
let jsonNode = %userId
|
|
assert jsonNode.kind == JString
|
|
assert jsonNode.getStr() == "user-42"
|
|
|
|
# Test deserialization
|
|
let jsonStr = "\"user-42\""
|
|
let parsedJson = !JsonNode.parse(jsonStr)
|
|
let result = UserId.fromJson(parsedJson)
|
|
assert result.isSuccess
|
|
assert int(!result) == 42
|
|
|
|
# Test in object context
|
|
type User {.serialize(mode = OptOut).} = object
|
|
id: UserId
|
|
name: string
|
|
|
|
let user = User(id: UserId(123), name: "John")
|
|
let userJson = %user
|
|
assert userJson.kind == JObject
|
|
assert userJson["id"].getStr() == "user-123"
|
|
assert userJson["name"].getStr() == "John"
|
|
|
|
# Test deserialization of object with custom type
|
|
let userJsonStr = """{"id":"user-123","name":"John"}"""
|
|
let userResult = User.fromJson(userJsonStr)
|
|
assert userResult.isSuccess
|
|
assert int((!userResult).id) == 123
|
|
assert (!userResult).name == "John"
|
|
```
|
|
|
|
## Deserialization API
|
|
|
|
nim-serde provides a type-safe way to convert JSON data back into Nim types.
|
|
|
|
### Basic Deserialization with `fromJson`
|
|
|
|
The `fromJson` function converts JSON strings or `JsonNode` objects to Nim types:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
import pkg/questionable/results
|
|
|
|
type Person = object
|
|
name: string
|
|
age: int
|
|
|
|
let jsonStr = """{"name":"John","age":30}"""
|
|
let result = Person.fromJson(jsonStr)
|
|
|
|
# Using the ! operator from questionable to extract the value
|
|
assert !result == Person(name: "John", age: 30)
|
|
```
|
|
|
|
### Error Handling
|
|
|
|
Deserialization returns a `Result` type from the `questionable` library, allowing for safe error handling:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
import pkg/questionable/results
|
|
|
|
type Person = object
|
|
name: string
|
|
age: int
|
|
|
|
let invalidJson = """{"name":"John","age":"thirty"}"""
|
|
let errorResult = Person.fromJson(invalidJson)
|
|
assert errorResult.isFailure
|
|
assert errorResult.error of UnexpectedKindError
|
|
```
|
|
|
|
### Parsing JSON with `JsonNode.parse`
|
|
|
|
To parse JSON string into a `JsonNode` tree instead of a deserializing to a concrete type, use `JsonNode.parse`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
import pkg/questionable/results
|
|
|
|
let jsonStr = """{"name":"John","age":30,"hobbies":["reading","coding"]}"""
|
|
let parseResult = JsonNode.parse(jsonStr)
|
|
assert parseResult.isSuccess
|
|
let jsonNode = !parseResult
|
|
assert jsonNode["name"].getStr == "John"
|
|
assert jsonNode["age"].getInt == 30
|
|
assert jsonNode["hobbies"].kind == JArray
|
|
assert jsonNode["hobbies"][0].getStr == "reading"
|
|
assert jsonNode["hobbies"][1].getStr == "coding"
|
|
```
|
|
|
|
### Deserialization Modes
|
|
|
|
nim-serde offers three modes to control how JSON is deserialized:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
import pkg/questionable/results
|
|
|
|
# OptOut mode (default for deserialization)
|
|
type PersonOptOut = object
|
|
name: string
|
|
age: int
|
|
|
|
let jsonOptOut = """{"name":"John","age":30,"address":"123 Main St"}"""
|
|
let resultOptOut = PersonOptOut.fromJson(jsonOptOut)
|
|
assert resultOptOut.isSuccess
|
|
assert !resultOptOut == PersonOptOut(name: "John", age: 30)
|
|
|
|
# OptIn mode
|
|
type PersonOptIn {.deserialize(mode = OptIn).} = object
|
|
name {.deserialize.}: string
|
|
age {.deserialize.}: int
|
|
address: string # Not deserialized by default in OptIn mode
|
|
|
|
let jsonOptIn = """{"name":"John","age":30,"address":"123 Main St"}"""
|
|
let resultOptIn = PersonOptIn.fromJson(jsonOptIn)
|
|
assert resultOptIn.isSuccess
|
|
assert (!resultOptIn).name == "John"
|
|
assert (!resultOptIn).age == 30
|
|
assert (!resultOptIn).address == "" # address is not deserialized
|
|
|
|
# Strict mode
|
|
type PersonStrict {.deserialize(mode = Strict).} = object
|
|
name: string
|
|
age: int
|
|
|
|
let jsonStrict = """{"name":"John","age":30}"""
|
|
let resultStrict = PersonStrict.fromJson(jsonStrict)
|
|
assert resultStrict.isSuccess
|
|
assert !resultStrict == PersonStrict(name: "John", age: 30)
|
|
|
|
# Strict mode with extra field (should fail)
|
|
let jsonStrictExtra = """{"name":"John","age":30,"address":"123 Main St"}"""
|
|
let resultStrictExtra = PersonStrict.fromJson(jsonStrictExtra)
|
|
assert resultStrictExtra.isFailure
|
|
```
|
|
|
|
### Field Customization for Deserialization
|
|
|
|
Fields can be customized with various options for deserialization:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
import pkg/questionable/results
|
|
|
|
type User = object
|
|
firstName {.deserialize(key = "first_name").}: string
|
|
lastName {.deserialize(key = "last_name").}: string
|
|
age: int
|
|
internalId {.deserialize(ignore = true).}: int
|
|
|
|
let userJsonStr = """{"first_name":"Jane","last_name":"Smith","age":25,"role":"admin"}"""
|
|
let result = User.fromJson(userJsonStr)
|
|
assert result.isSuccess
|
|
assert (!result).firstName == "Jane"
|
|
assert (!result).lastName == "Smith"
|
|
assert (!result).age == 25
|
|
assert (!result).internalId == 0 # Default value, not deserialized
|
|
```
|
|
|
|
## Using as a Drop-in Replacement for std/json
|
|
|
|
nim-serde can be used as a drop-in replacement for the standard library's `json` module with improved exception handling.
|
|
|
|
```nim
|
|
# Instead of:
|
|
# import std/json
|
|
import pkg/serde/json
|
|
import pkg/questionable/results
|
|
|
|
# Using nim-serde's JSON API which is compatible with std/json
|
|
let jsonNode = %* {
|
|
"name": "John",
|
|
"age": 30,
|
|
"isActive": true,
|
|
"hobbies": ["reading", "swimming"]
|
|
}
|
|
|
|
# Accessing JSON fields using the same API as std/json
|
|
assert jsonNode.kind == JObject
|
|
assert jsonNode["name"].getStr == "John"
|
|
assert jsonNode["age"].getInt == 30
|
|
assert jsonNode["isActive"].getBool == true
|
|
assert jsonNode["hobbies"].kind == JArray
|
|
assert jsonNode["hobbies"][0].getStr == "reading"
|
|
assert jsonNode["hobbies"][1].getStr == "swimming"
|
|
|
|
# Converting JSON to string
|
|
let jsonStr = $jsonNode
|
|
|
|
# Parsing JSON from string with better error handling
|
|
let parsedResult = JsonNode.parse(jsonStr)
|
|
assert parsedResult.isSuccess
|
|
let parsedNode = !parsedResult
|
|
assert parsedNode.kind == JObject
|
|
assert parsedNode["name"].getStr == "John"
|
|
|
|
# Pretty printing
|
|
let prettyJson = pretty(jsonNode)
|
|
```
|
|
|
|
## Implementation Details
|
|
|
|
The JSON serialization in nim-serde is based on the `%` operator pattern:
|
|
|
|
1. The `%` operator converts values to `JsonNode` objects
|
|
2. Various overloads handle different types (primitives, objects, collections)
|
|
3. The `toJson` function converts the `JsonNode` to a string
|
|
|
|
This approach makes it easy to extend with custom types by defining your own `%` operator overloads.
|
|
|
|
For deserialization, the library uses compile-time reflection to map JSON fields to object fields, respecting the configuration provided by pragmas.
|