mirror of
https://github.com/logos-storage/nim-serde.git
synced 2026-01-02 21:53:07 +00:00
cdba47becf
* fix: force symbol resolution for types that serde de/serializes Force symbols into scope when mixins are used with generic overloads. When mixins are used with generic overloads, the overloaded symbols in scope of the mixin are evaluated from the perspective of the mixin. This creates issues in downstream modules that may inadvertantly dispatch *only* to the symbols in the scope of the mixin, even when the module with the wrong symbol overloads is not imported. By forcing the compiler to use symbols for types handled by serde, we can be sure that these symbols are available to downstream modules. We can also be sure that these `fromJson` symbols can be overloaded where needed. * remove enum forced scoping Forcing a scoping for a particular enum type would only resolve that type and not all enum types. * Add mixin + generic overloads as known issue to README * try to fix URL reference to deserializer.nim
413 lines
13 KiB
Markdown
413 lines
13 KiB
Markdown
# nim-serde
|
|
|
|
Easy-to-use json serialization capabilities, and a drop-in replacement for `std/json`.
|
|
|
|
## Quick examples
|
|
|
|
Opt-in serialization by default:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType = object
|
|
field1 {.serialize.}: bool
|
|
field2: bool
|
|
|
|
assert MyType(field1: true, field2: true).toJson == """{"field1":true}"""
|
|
```
|
|
|
|
Opt-out deserialization by default:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
# All fields deserialized, as none are ignored
|
|
type MyType1 = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
let jsn1 = """{
|
|
"field1": true,
|
|
"field2": true
|
|
}"""
|
|
assert !MyType1.fromJson(jsn1) == MyType1(field1: true, field2: true)
|
|
|
|
# Don't deserialize ignored fields in OptOut mode
|
|
type MyType2 = object
|
|
field1 {.deserialize(ignore=true).}: bool
|
|
field2: bool
|
|
|
|
let jsn2 = """{
|
|
"field1": true,
|
|
"field2": true,
|
|
"extra": "extra fields don't error in OptOut mode"
|
|
}"""
|
|
assert !MyType2.fromJson(jsn2) == MyType2(field1: false, field2: true)
|
|
|
|
# Note, the ! operator is part of https://github.com/codex-storage/questionable, which retrieves a value if set
|
|
```
|
|
|
|
Serialize all fields of a type (OptOut mode):
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.serialize.} = object
|
|
field1: int
|
|
field2: int
|
|
|
|
assert MyType(field1: 1, field2: 2).toJson == """{"field1":1,"field2":2}"""
|
|
```
|
|
|
|
Alias field names in both directions!
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.serialize.} = object
|
|
field1 {.serialize("othername"),deserialize("takesprecedence").}: int
|
|
field2: int
|
|
|
|
assert MyType(field1: 1, field2: 2).toJson == """{"othername":1,"field2":2}"""
|
|
let jsn = """{
|
|
"othername": 1,
|
|
"field2": 2,
|
|
"takesprecedence": 3
|
|
}"""
|
|
assert !MyType.fromJson(jsn) == MyType(field1: 3, field2: 2)
|
|
```
|
|
|
|
Supports strict mode, where type fields and json fields must match
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.deserialize(mode=Strict).} = object
|
|
field1: int
|
|
field2: int
|
|
|
|
let jsn = """{
|
|
"field1": 1,
|
|
"field2": 2,
|
|
"extra": 3
|
|
}"""
|
|
|
|
let res = MyType.fromJson(jsn)
|
|
assert res.isFailure
|
|
assert res.error of SerdeError
|
|
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"
|
|
```
|
|
|
|
## Serde modes
|
|
|
|
`nim-serde` uses three different modes to control de/serialization:
|
|
|
|
```nim
|
|
OptIn
|
|
OptOut
|
|
Strict
|
|
```
|
|
|
|
Modes can be set in the `{.serialize.}` and/or `{.deserialize.}` pragmas on type
|
|
definitions. Each mode has a different meaning depending on if the type is being
|
|
serialized or deserialized. Modes can be set by setting `mode` in the `serialize` or
|
|
`deserialize` pragma annotation, eg:
|
|
|
|
```nim
|
|
type MyType {.serialize(mode=Strict).} = object
|
|
field1: bool
|
|
field2: bool
|
|
```
|
|
|
|
### Modes reference
|
|
|
|
| | serialize | deserialize |
|
|
|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
| `SerdeMode.OptOut` | All object fields will be serialized, except fields marked with `{.serialize(ignore=true).}`. | All json keys will be deserialized, except fields marked with `{.deserialize(ignore=true).}`. No error if extra json fields exist. |
|
|
| `SerdeMode.OptIn` | Only fields marked with `{.serialize.}` will be serialized. Fields marked with `{.serialize(ignore=true).}` will not be serialized. | Only fields marked with `{.deserialize.}` will be deserialized. Fields marked with `{.deserialize(ignore=true).}` will not be deserialized. A `SerdeError` error is raised if the field is missing in json. |
|
|
| `SerdeMode.Strict` | All object fields will be serialized, regardless if the field is marked with `{.serialize(ignore=true).}`. | Object fields and json fields must match exactly, otherwise a `SerdeError` is raised. |
|
|
|
|
## Default modes
|
|
|
|
`nim-serde` will de/serialize types if they are not annotated with `serialize` or
|
|
`deserialize`, but will assume a default mode. By default, with no pragmas specified,
|
|
`serde` will always serialize in `OptIn` mode, meaning any fields to b Additionally, if
|
|
the types are annotated, but a mode is not specified, `serde` will assume a (possibly
|
|
different) default mode.
|
|
|
|
```nim
|
|
# Type is not annotated
|
|
# A default mode of OptIn (for serialize) and OptOut (for deserialize) is assumed.
|
|
|
|
type MyObj = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
# Type is annotated, but mode not specified
|
|
# A default mode of OptOut is assumed for both serialize and deserialize.
|
|
|
|
type MyObj {.serialize, deserialize.} = object
|
|
field1: bool
|
|
field2: bool
|
|
```
|
|
|
|
### Default mode reference
|
|
|
|
| | serialize | deserialize |
|
|
|:------------------------------|:----------|:------------|
|
|
| Default (no pragma) | `OptIn` | `OptOut` |
|
|
| Default (pragma, but no mode) | `OptOut` | `OptOut` |
|
|
|
|
## Serde field options
|
|
Type fields can be annotated with `{.serialize.}` and `{.deserialize.}` and properties
|
|
can be set on these pragmas, determining de/serialization behavior.
|
|
|
|
For example,
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type
|
|
Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
|
|
id {.serialize(ignore=true), deserialize(key="personid").}: int
|
|
name: string
|
|
birthYear: int
|
|
address: string
|
|
phone: string
|
|
|
|
let person = Person(
|
|
name: "Lloyd Christmas",
|
|
birthYear: 1970,
|
|
address: "123 Sesame Street, Providence, Rhode Island 12345",
|
|
phone: "555-905-justgivemethedamnnumber!⛽️🔥")
|
|
|
|
let createRequest = """{
|
|
"name": "Lloyd Christmas",
|
|
"birthYear": 1970,
|
|
"address": "123 Sesame Street, Providence, Rhode Island 12345",
|
|
"phone": "555-905-justgivemethedamnnumber!⛽️🔥"
|
|
}"""
|
|
assert person.toJson(pretty=true) == createRequest
|
|
|
|
let createResponse = """{
|
|
"personid": 1,
|
|
"name": "Lloyd Christmas",
|
|
"birthYear": 1970,
|
|
"address": "123 Sesame Street, Providence, Rhode Island 12345",
|
|
"phone": "555-905-justgivemethedamnnumber!⛽️🔥"
|
|
}"""
|
|
assert !Person.fromJson(createResponse) == Person(id: 1)
|
|
```
|
|
|
|
### `key`
|
|
Specifying a `key`, will alias the field name. When seriazlizing, json will be written
|
|
with `key` instead of the field name. When deserializing, the json must contain `key`
|
|
for the field to be deserialized.
|
|
|
|
### `ignore`
|
|
Specifying `ignore`, will prevent de/serialization on the field.
|
|
|
|
### Serde field options reference
|
|
|
|
| | serialize | deserialize |
|
|
|:---------|:-----------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------|
|
|
| `key` | aliases the field name in json | deserializes the field if json contains `key` |
|
|
| `ignore` | <li>**OptOut:** field not serialized</li><li>**OptIn:** field not serialized</li><li>**Strict:** field serialized</li> | <li>**OptOut:** field not deserialized</li><li>**OptIn:** field not deserialized</li><li>**Strict:** field deserialized</li> |
|
|
|
|
|
|
## Deserialization
|
|
|
|
`serde` deserializes using `fromJson`, and in all instances returns `Result[T,
|
|
CatchableError]`, where `T` is the type being deserialized. For example:
|
|
|
|
```nim
|
|
type MyType = object
|
|
field1: bool
|
|
field2: bool
|
|
|
|
let jsn1 = """{
|
|
"field1": true,
|
|
"field2": true
|
|
}"""
|
|
|
|
assert !MyType.fromJson(jsn1) == MyType(field1: true, field2: true)
|
|
```
|
|
|
|
If there was an error during deserialization, the result of `fromJson` will contain it:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.deserialize(mode=Strict).} = object
|
|
field1: int
|
|
field2: int
|
|
|
|
let jsn = """{
|
|
"field1": 1,
|
|
"field2": 2,
|
|
"extra": 3
|
|
}"""
|
|
|
|
let res = MyType.fromJson(jsn)
|
|
assert res.isFailure
|
|
assert res.error of SerdeError
|
|
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"
|
|
```
|
|
|
|
## Custom types
|
|
|
|
If `serde` can't de/serialize a custom type, de/serialization can be supported by
|
|
overloading `%` and `fromJson`. For example:
|
|
|
|
```nim
|
|
type
|
|
Address* = distinct array[20, byte]
|
|
SerializationError* = object of CatchableError
|
|
|
|
func `%`*(address: Address): JsonNode =
|
|
%($address)
|
|
|
|
func fromJson(_: type Address, json: JsonNode): ?!Address =
|
|
expectJsonKind(Address, JString, json)
|
|
without address =? Address.init(json.getStr), error:
|
|
return failure newException(SerializationError,
|
|
"Failed to convert '" & $json & "' to Address: " & error.msg)
|
|
success address
|
|
```
|
|
|
|
## Serializing to string (`toJson`)
|
|
|
|
`toJson` is a shortcut for serializing an object into its serialized string
|
|
representation:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
type MyType {.serialize.} = object
|
|
field1: string
|
|
field2: bool
|
|
|
|
let mt = MyType(field1: "hw", field2: true)
|
|
assert mt.toJson == """{"field1":"hw","field2":true}"""
|
|
```
|
|
|
|
This comes in handy, for example, when sending API responses:
|
|
|
|
```nim
|
|
let availability = getAvailability(...)
|
|
return RestApiResponse.response(availability.toJson,
|
|
contentType="application/json")
|
|
```
|
|
|
|
## `std/json` drop-in replacment
|
|
|
|
`nim-serde` can be used as a drop-in replacement for the [standard library's `json`
|
|
module](https://nim-lang.org/docs/json.html), with a few notable improvements.
|
|
|
|
Instead of importing `std/json` into your application, `pkg/serde/json` can be imported
|
|
instead:
|
|
|
|
```diff
|
|
- import std/json
|
|
+ import pkg/serde/json
|
|
```
|
|
|
|
As with `std/json`, `%` can be used to serialize a type into a `JsonNode`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
assert %"hello" == newJString("hello")
|
|
```
|
|
|
|
And `%*` can be used to serialize objects:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
let expected = newJObject()
|
|
expected["hello"] = newJString("world")
|
|
assert %*{"hello": "world"} == expected
|
|
```
|
|
|
|
As well, serialization of types can be overridden, and serialization of custom types can
|
|
be introduced. Here, we are overriding the serialization of `int`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
func `%`(i: int): JsonNode =
|
|
newJInt(i + 1)
|
|
|
|
assert 1.toJson == "2"
|
|
```
|
|
|
|
## `parseJson` and exception tracking
|
|
|
|
Unfortunately, `std/json`'s `parseJson` can raise an `Exception`, so proper exception
|
|
tracking breaks, eg
|
|
|
|
```nim
|
|
|
|
## Fails to compile:
|
|
## Error: parseJson(me, false, false) can raise an unlisted exception: Exception
|
|
|
|
import std/json
|
|
|
|
{.push raises:[].}
|
|
|
|
type
|
|
MyAppError = object of CatchableError
|
|
|
|
proc parseMe(me: string): JsonNode =
|
|
try:
|
|
return me.parseJson
|
|
except CatchableError as error:
|
|
raise newException(MyAppError, error.msg)
|
|
|
|
assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }
|
|
```
|
|
|
|
This is due to `std/json`'s `parseJson` incorrectly raising `Exception`. This can be
|
|
worked around by instead importing `serde` and calling its `JsonNode.parse` routine.
|
|
Note that `serde`'s `JsonNode.parse` returns a `Result[JsonNode, CatchableError]`
|
|
instead of just a plain `JsonNode` object as in `std/json`'s `parseJson`:
|
|
|
|
```nim
|
|
import pkg/serde/json
|
|
|
|
{.push raises:[].}
|
|
|
|
type
|
|
MyAppError = object of CatchableError
|
|
|
|
proc parseMe(me: string): JsonNode {.raises: [MyAppError].} =
|
|
without parsed =? JsonNode.parse(me), error:
|
|
raise newException(MyAppError, error.msg)
|
|
parsed
|
|
|
|
assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }
|
|
```
|
|
|
|
## Known issues
|
|
|
|
There is a known issue when using mixins with generic overloaded procs like
|
|
`fromJson`. At the time of mixin call, only the `fromJson` overloads in scope of
|
|
the called mixin are available to be dispatched at runtime. There could be other
|
|
`fromJson` overloads declared in other modules, but are not in scope at the time
|
|
the mixin was called. Therefore, anytime `fromJson` is called targeting a
|
|
declared overload, it may or may not be dispatchable. This can be worked around
|
|
by forcing the `fromJson` overload into scope at compile time. For example, in
|
|
your application where the `fromJson` overload is defined, at the bottom of the
|
|
module add:
|
|
```nim
|
|
static: MyType.fromJson("")
|
|
```
|
|
This will ensure that the `MyType.fromJson` overload is dispatchable.
|
|
|
|
The basic types that serde supports should already have their overloads forced
|
|
in scope in [the `deserializer`
|
|
module](./serde/json/deserializer.nim#L340-L356).
|
|
|
|
For an illustration of the problem, please see this [narrow example](https://github.com/gmega/serialization-bug/tree/main/narrow) by [@gmega](https://github.com/gmega).
|