mirror of
https://github.com/logos-storage/nim-serde.git
synced 2026-01-02 13:43:06 +00:00
413 lines
13 KiB
Markdown
413 lines
13 KiB
Markdown
# nim-serde
|
|
|
|
Easy-to-use json serialization capabilities, and a drop-in replacement for `std/json`.
|
|
|
|
## Quick examples
|
|
|
|
Opt-in serialization by default:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType = object
|
|
field1 {.serialize.}: bool
|
|
field2: bool
|
|
|
|
assert MyType(field1: true, field2: true).toJson == """{"field1":true}"""
|
|
```
|
|
|
|
Opt-out deserialization by default:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
# All fields deserialized, as none are ignored
|
|
type MyType1 = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
let jsn1 = """{
|
|
"field1": true,
|
|
"field2": true
|
|
}"""
|
|
assert !MyType1.fromJson(jsn1) == MyType1(field1: true, field2: true)
|
|
|
|
# Don't deserialize ignored fields in OptOut mode
|
|
type MyType2 = object
|
|
field1 {.deserialize(ignore=true).}: bool
|
|
field2: bool
|
|
|
|
let jsn2 = """{
|
|
"field1": true,
|
|
"field2": true,
|
|
"extra": "extra fields don't error in OptOut mode"
|
|
}"""
|
|
assert !MyType2.fromJson(jsn2) == MyType2(field1: false, field2: true)
|
|
|
|
# Note, the ! operator is part of https://github.com/logos-storage/questionable, which retrieves a value if set
|
|
```
|
|
|
|
Serialize all fields of a type (OptOut mode):
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.serialize.} = object
|
|
field1: int
|
|
field2: int
|
|
|
|
assert MyType(field1: 1, field2: 2).toJson == """{"field1":1,"field2":2}"""
|
|
```
|
|
|
|
Alias field names in both directions!
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.serialize.} = object
|
|
field1 {.serialize("othername"),deserialize("takesprecedence").}: int
|
|
field2: int
|
|
|
|
assert MyType(field1: 1, field2: 2).toJson == """{"othername":1,"field2":2}"""
|
|
let jsn = """{
|
|
"othername": 1,
|
|
"field2": 2,
|
|
"takesprecedence": 3
|
|
}"""
|
|
assert !MyType.fromJson(jsn) == MyType(field1: 3, field2: 2)
|
|
```
|
|
|
|
Supports strict mode, where type fields and json fields must match
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.deserialize(mode=Strict).} = object
|
|
field1: int
|
|
field2: int
|
|
|
|
let jsn = """{
|
|
"field1": 1,
|
|
"field2": 2,
|
|
"extra": 3
|
|
}"""
|
|
|
|
let res = MyType.fromJson(jsn)
|
|
assert res.isFailure
|
|
assert res.error of SerdeError
|
|
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"
|
|
```
|
|
|
|
## Serde modes
|
|
|
|
`nim-serde` uses three different modes to control de/serialization:
|
|
|
|
```nim
|
|
OptIn
|
|
OptOut
|
|
Strict
|
|
```
|
|
|
|
Modes can be set in the `{.serialize.}` and/or `{.deserialize.}` pragmas on type
|
|
definitions. Each mode has a different meaning depending on if the type is being
|
|
serialized or deserialized. Modes can be set by setting `mode` in the `serialize` or
|
|
`deserialize` pragma annotation, eg:
|
|
|
|
```nim
|
|
type MyType {.serialize(mode=Strict).} = object
|
|
field1: bool
|
|
field2: bool
|
|
```
|
|
|
|
### Modes reference
|
|
|
|
| | serialize | deserialize |
|
|
|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
| `SerdeMode.OptOut` | All object fields will be serialized, except fields marked with `{.serialize(ignore=true).}`. | All json keys will be deserialized, except fields marked with `{.deserialize(ignore=true).}`. No error if extra json fields exist. |
|
|
| `SerdeMode.OptIn` | Only fields marked with `{.serialize.}` will be serialized. Fields marked with `{.serialize(ignore=true).}` will not be serialized. | Only fields marked with `{.deserialize.}` will be deserialized. Fields marked with `{.deserialize(ignore=true).}` will not be deserialized. A `SerdeError` error is raised if the field is missing in json. |
|
|
| `SerdeMode.Strict` | All object fields will be serialized, regardless if the field is marked with `{.serialize(ignore=true).}`. | Object fields and json fields must match exactly, otherwise a `SerdeError` is raised. |
|
|
|
|
## Default modes
|
|
|
|
`nim-serde` will de/serialize types if they are not annotated with `serialize` or
|
|
`deserialize`, but will assume a default mode. By default, with no pragmas specified,
|
|
`serde` will always serialize in `OptIn` mode, meaning any fields to b Additionally, if
|
|
the types are annotated, but a mode is not specified, `serde` will assume a (possibly
|
|
different) default mode.
|
|
|
|
```nim
|
|
# Type is not annotated
|
|
# A default mode of OptIn (for serialize) and OptOut (for deserialize) is assumed.
|
|
|
|
type MyObj = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
# Type is annotated, but mode not specified
|
|
# A default mode of OptOut is assumed for both serialize and deserialize.
|
|
|
|
type MyObj {.serialize, deserialize.} = object
|
|
field1: bool
|
|
field2: bool
|
|
```
|
|
|
|
### Default mode reference
|
|
|
|
| | serialize | deserialize |
|
|
|:------------------------------|:----------|:------------|
|
|
| Default (no pragma) | `OptIn` | `OptOut` |
|
|
| Default (pragma, but no mode) | `OptOut` | `OptOut` |
|
|
|
|
## Serde field options
|
|
Type fields can be annotated with `{.serialize.}` and `{.deserialize.}` and properties
|
|
can be set on these pragmas, determining de/serialization behavior.
|
|
|
|
For example,
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type
|
|
Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
|
|
id {.serialize(ignore=true), deserialize(key="personid").}: int
|
|
name: string
|
|
birthYear: int
|
|
address: string
|
|
phone: string
|
|
|
|
let person = Person(
|
|
name: "Lloyd Christmas",
|
|
birthYear: 1970,
|
|
address: "123 Sesame Street, Providence, Rhode Island 12345",
|
|
phone: "555-905-justgivemethedamnnumber!⛽️🔥")
|
|
|
|
let createRequest = """{
|
|
"name": "Lloyd Christmas",
|
|
"birthYear": 1970,
|
|
"address": "123 Sesame Street, Providence, Rhode Island 12345",
|
|
"phone": "555-905-justgivemethedamnnumber!⛽️🔥"
|
|
}"""
|
|
assert person.toJson(pretty=true) == createRequest
|
|
|
|
let createResponse = """{
|
|
"personid": 1,
|
|
"name": "Lloyd Christmas",
|
|
"birthYear": 1970,
|
|
"address": "123 Sesame Street, Providence, Rhode Island 12345",
|
|
"phone": "555-905-justgivemethedamnnumber!⛽️🔥"
|
|
}"""
|
|
assert !Person.fromJson(createResponse) == Person(id: 1)
|
|
```
|
|
|
|
### `key`
|
|
Specifying a `key`, will alias the field name. When seriazlizing, json will be written
|
|
with `key` instead of the field name. When deserializing, the json must contain `key`
|
|
for the field to be deserialized.
|
|
|
|
### `ignore`
|
|
Specifying `ignore`, will prevent de/serialization on the field.
|
|
|
|
### Serde field options reference
|
|
|
|
| | serialize | deserialize |
|
|
|:---------|:-----------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------|
|
|
| `key` | aliases the field name in json | deserializes the field if json contains `key` |
|
|
| `ignore` | <li>**OptOut:** field not serialized</li><li>**OptIn:** field not serialized</li><li>**Strict:** field serialized</li> | <li>**OptOut:** field not deserialized</li><li>**OptIn:** field not deserialized</li><li>**Strict:** field deserialized</li> |
|
|
|
|
|
|
## Deserialization
|
|
|
|
`serde` deserializes using `fromJson`, and in all instances returns `Result[T,
|
|
CatchableError]`, where `T` is the type being deserialized. For example:
|
|
|
|
```nim
|
|
type MyType = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
let jsn1 = """{
|
|
"field1": true,
|
|
"field2": true
|
|
}"""
|
|
|
|
assert !MyType.fromJson(jsn1) == MyType(field1: true, field2: true)
|
|
```
|
|
|
|
If there was an error during deserialization, the result of `fromJson` will contain it:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.deserialize(mode=Strict).} = object
|
|
field1: int
|
|
field2: int
|
|
|
|
let jsn = """{
|
|
"field1": 1,
|
|
"field2": 2,
|
|
"extra": 3
|
|
}"""
|
|
|
|
let res = MyType.fromJson(jsn)
|
|
assert res.isFailure
|
|
assert res.error of SerdeError
|
|
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"
|
|
```
|
|
|
|
## Custom types
|
|
|
|
If `serde` can't de/serialize a custom type, de/serialization can be supported by
|
|
overloading `%` and `fromJson`. For example:
|
|
|
|
```nim
|
|
type
|
|
Address* = distinct array[20, byte]
|
|
SerializationError* = object of CatchableError
|
|
|
|
func `%`*(address: Address): JsonNode =
|
|
%($address)
|
|
|
|
func fromJson(_: type Address, json: JsonNode): ?!Address =
|
|
expectJsonKind(Address, JString, json)
|
|
without address =? Address.init(json.getStr), error:
|
|
return failure newException(SerializationError,
|
|
"Failed to convert '" & $json & "' to Address: " & error.msg)
|
|
success address
|
|
```
|
|
|
|
## Serializing to string (`toJson`)
|
|
|
|
`toJson` is a shortcut for serializing an object into its serialized string
|
|
representation:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.serialize.} = object
|
|
field1: string
|
|
field2: bool
|
|
|
|
let mt = MyType(field1: "hw", field2: true)
|
|
assert mt.toJson == """{"field1":"hw","field2":true}"""
|
|
```
|
|
|
|
This comes in handy, for example, when sending API responses:
|
|
|
|
```nim
|
|
let availability = getAvailability(...)
|
|
return RestApiResponse.response(availability.toJson,
|
|
contentType="application/json")
|
|
```
|
|
|
|
## `std/json` drop-in replacment
|
|
|
|
`nim-serde` can be used as a drop-in replacement for the [standard library's `json`
|
|
module](https://nim-lang.org/docs/json.html), with a few notable improvements.
|
|
|
|
Instead of importing `std/json` into your application, `pkg/serde/json` can be imported
|
|
instead:
|
|
|
|
```diff
|
|
- import std/json
|
|
+ import pkg/serde/json
|
|
```
|
|
|
|
As with `std/json`, `%` can be used to serialize a type into a `JsonNode`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
assert %"hello" == newJString("hello")
|
|
```
|
|
|
|
And `%*` can be used to serialize objects:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
let expected = newJObject()
|
|
expected["hello"] = newJString("world")
|
|
assert %*{"hello": "world"} == expected
|
|
```
|
|
|
|
As well, serialization of types can be overridden, and serialization of custom types can
|
|
be introduced. Here, we are overriding the serialization of `int`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
func `%`(i: int): JsonNode =
|
|
newJInt(i + 1)
|
|
|
|
assert 1.toJson == "2"
|
|
```
|
|
|
|
## `parseJson` and exception tracking
|
|
|
|
Unfortunately, `std/json`'s `parseJson` can raise an `Exception`, so proper exception
|
|
tracking breaks, eg
|
|
|
|
```nim
|
|
|
|
## Fails to compile:
|
|
## Error: parseJson(me, false, false) can raise an unlisted exception: Exception
|
|
|
|
import std/json
|
|
|
|
{.push raises:[].}
|
|
|
|
type
|
|
MyAppError = object of CatchableError
|
|
|
|
proc parseMe(me: string): JsonNode =
|
|
try:
|
|
return me.parseJson
|
|
except CatchableError as error:
|
|
raise newException(MyAppError, error.msg)
|
|
|
|
assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }
|
|
```
|
|
|
|
This is due to `std/json`'s `parseJson` incorrectly raising `Exception`. This can be
|
|
worked around by instead importing `serde` and calling its `JsonNode.parse` routine.
|
|
Note that `serde`'s `JsonNode.parse` returns a `Result[JsonNode, CatchableError]`
|
|
instead of just a plain `JsonNode` object as in `std/json`'s `parseJson`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
{.push raises:[].}
|
|
|
|
type
|
|
MyAppError = object of CatchableError
|
|
|
|
proc parseMe(me: string): JsonNode {.raises: [MyAppError].} =
|
|
without parsed =? JsonNode.parse(me), error:
|
|
raise newException(MyAppError, error.msg)
|
|
parsed
|
|
|
|
assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }
|
|
```
|
|
|
|
## Known issues
|
|
|
|
There is a known issue when using mixins with generic overloaded procs like
|
|
`fromJson`. At the time of mixin call, only the `fromJson` overloads in scope of
|
|
the called mixin are available to be dispatched at runtime. There could be other
|
|
`fromJson` overloads declared in other modules, but are not in scope at the time
|
|
the mixin was called. Therefore, anytime `fromJson` is called targeting a
|
|
declared overload, it may or may not be dispatchable. This can be worked around
|
|
by forcing the `fromJson` overload into scope at compile time. For example, in
|
|
your application where the `fromJson` overload is defined, at the bottom of the
|
|
module add:
|
|
```nim
|
|
static: MyType.fromJson("")
|
|
```
|
|
This will ensure that the `MyType.fromJson` overload is dispatchable.
|
|
|
|
The basic types that serde supports should already have their overloads forced
|
|
in scope in [the `deserializer`
|
|
module](./serde/json/deserializer.nim#L340-L356).
|
|
|
|
For an illustration of the problem, please see this [narrow example](https://github.com/gmega/serialization-bug/tree/main/narrow) by [@gmega](https://github.com/gmega).
|