nim-serde/README.md

# nim-serde

Easy-to-use json serialization capabilities, and a drop-in replacement for `std/json`.

## Quick examples

Opt-in serialization by default:

```nim
import pkg/serde/json

type MyType = object
  field1 {.serialize.}: bool
  field2: bool

assert MyType(field1: true, field2: true).toJson == """{"field1":true}"""
```

Opt-out deserialization by default:

```nim
import pkg/serde/json

# All fields deserialized, as none are ignored
type MyType1 = object
  field1: bool
  field2: bool

let jsn1 = """{
                "field1": true,
                "field2": true
              }"""
assert !MyType1.fromJson(jsn1) == MyType1(field1: true, field2: true)

# Don't deserialize ignored fields in OptOut mode
type MyType2 = object
  field1 {.deserialize(ignore=true).}: bool
  field2: bool

let jsn2 = """{
                "field1": true,
                "field2": true,
                "extra": "extra fields don't error in OptOut mode"
              }"""
assert !MyType2.fromJson(jsn2) == MyType2(field1: false, field2: true)

# Note, the ! operator is part of https://github.com/codex-storage/questionable, which retrieves a value if set
```

Serialize all fields of a type (OptOut mode):

```nim
import pkg/serde/json

type MyType {.serialize.} = object
  field1: int
  field2: int

assert MyType(field1: 1, field2: 2).toJson == """{"field1":1,"field2":2}"""
```

Alias field names in both directions!

```nim
import pkg/serde/json

type MyType {.serialize.} = object
  field1 {.serialize("othername"),deserialize("takesprecedence").}: int
  field2: int

assert MyType(field1: 1, field2: 2).toJson == """{"othername":1,"field2":2}"""
let jsn = """{
                "othername":       1,
                "field2":          2,
                "takesprecedence": 3
              }"""
assert !MyType.fromJson(jsn) == MyType(field1: 3, field2: 2)
```

Supports strict mode, where type fields and json fields must match

```nim
import pkg/serde/json

type MyType {.deserialize(mode=Strict).} = object
  field1: int
  field2: int

let jsn = """{
                "field1": 1,
                "field2": 2,
                "extra":  3
              }"""

let res = MyType.fromJson(jsn)
assert res.isFailure
assert res.error of SerdeError
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"
```

## Serde modes

`nim-serde` uses three different modes to control de/serialization:

```nim
OptIn
OptOut
Strict
```

Modes can be set in the `{.serialize.}` and/or `{.deserialize.}` pragmas on type
definitions. Each mode has a different meaning depending on if the type is being
serialized or deserialized. Modes can be set by setting `mode` in the `serialize` or
`deserialize` pragma annotation, eg:

```nim
type MyType {.serialize(mode=Strict).} = object
  field1: bool
  field2: bool
```

### Modes reference

|                    | serialize                                                                                                                           | deserialize                                                                                                                                                                                                 |
|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `SerdeMode.OptOut` | All object fields will be serialized, except fields marked with `{.serialize(ignore=true).}`.                                       | All json keys will be deserialized, except fields marked with `{.deserialize(ignore=true).}`. No error if extra json fields exist.                                                                          |
| `SerdeMode.OptIn`  | Only fields marked with `{.serialize.}` will be serialized. Fields marked with `{.serialize(ignore=true).}` will not be serialized. | Only fields marked with `{.deserialize.}` will be deserialized. Fields marked with `{.deserialize(ignore=true).}` will not be deserialized. A `SerdeError` error is raised if the field is missing in json. |
| `SerdeMode.Strict` | All object fields will be serialized, regardless if the field is marked with `{.serialize(ignore=true).}`.                          | Object fields and json fields must match exactly, otherwise a `SerdeError` is raised.                                                                                                                       |

## Default modes

`nim-serde` will de/serialize types if they are not annotated with `serialize` or
`deserialize`, but will assume a default mode. By default, with no pragmas specified,
`serde` will always serialize in `OptIn` mode, meaning any fields to b Additionally, if
the types are annotated, but a mode is not specified, `serde` will assume a (possibly
different) default mode.

```nim
# Type is not annotated
# A default mode of OptIn (for serialize) and OptOut (for deserialize) is assumed.

type MyObj = object
  field1: bool
  field2: bool

# Type is annotated, but mode not specified
# A default mode of OptOut is assumed for both serialize and deserialize.

type MyObj {.serialize, deserialize.} = object
  field1: bool
  field2: bool
```

### Default mode reference

|                               | serialize | deserialize |
|:------------------------------|:----------|:------------|
| Default (no pragma)           | `OptIn`   | `OptOut`    |
| Default (pragma, but no mode) | `OptOut`  | `OptOut`    |

## Serde field options
Type fields can be annotated with `{.serialize.}` and `{.deserialize.}` and properties
can be set on these pragmas, determining de/serialization behavior.

For example,

```nim
import pkg/serde/json

type
  Person {.serialize(mode=OptOut), deserialize(mode=OptIn).} = object
    id {.serialize(ignore=true), deserialize(key="personid").}: int
    name: string
    birthYear: int
    address: string
    phone: string

let person = Person(
              name: "Lloyd Christmas",
              birthYear: 1970,
              address: "123 Sesame Street, Providence, Rhode Island  12345",
              phone: "555-905-justgivemethedamnnumber!⛽️🔥")

let createRequest = """{
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert person.toJson(pretty=true) == createRequest

let createResponse = """{
  "personid": 1,
  "name": "Lloyd Christmas",
  "birthYear": 1970,
  "address": "123 Sesame Street, Providence, Rhode Island  12345",
  "phone": "555-905-justgivemethedamnnumber!⛽️🔥"
}"""
assert !Person.fromJson(createResponse) == Person(id: 1)
```

### `key`
Specifying a `key`, will alias the field name. When seriazlizing, json will be written
with `key` instead of the field name. When deserializing, the json must contain `key`
for the field to be deserialized.

### `ignore`
Specifying `ignore`, will prevent de/serialization on the field.

### Serde field options reference

|          | serialize                                                                                                  | deserialize                                                                                                      |
|:---------|:-----------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------|
| `key`    | aliases the field name in json                                                                             | deserializes the field if json contains `key`                                                                    |
| `ignore` | <li>**OptOut:** field not serialized</li><li>**OptIn:** field not serialized</li><li>**Strict:** field serialized</li> | <li>**OptOut:** field not deserialized</li><li>**OptIn:** field not deserialized</li><li>**Strict:** field deserialized</li> |


## Deserialization

`serde` deserializes using `fromJson`, and in all instances returns `Result[T,
CatchableError]`, where `T` is the type being deserialized. For example:

```nim
type MyType = object
 field1: bool
 field2: bool

let jsn1 = """{
               "field1": true,
               "field2": true
             }"""

assert !MyType.fromJson(jsn1) == MyType(field1: true, field2: true)
```

If there was an error during deserialization, the result of `fromJson` will contain it:

```nim
import pkg/serde/json

type MyType {.deserialize(mode=Strict).} = object
  field1: int
  field2: int

let jsn = """{
                "field1": 1,
                "field2": 2,
                "extra":  3
              }"""

let res = MyType.fromJson(jsn)
assert res.isFailure
assert res.error of SerdeError
assert res.error.msg == "json field(s) missing in object: {\"extra\"}"
```

## Custom types

If `serde` can't de/serialize a custom type, de/serialization can be supported by
overloading `%` and `fromJson`. For example:

```nim
type
  Address* = distinct array[20, byte]
  SerializationError* = object of CatchableError

func `%`*(address: Address): JsonNode =
  %($address)

func fromJson(_: type Address, json: JsonNode): ?!Address =
  expectJsonKind(Address, JString, json)
  without address =? Address.init(json.getStr), error:
    return failure newException(SerializationError,
      "Failed to convert '" & $json & "' to Address: " & error.msg)
  success address
```

## Serializing to string (`toJson`)

`toJson` is a shortcut for serializing an object into its serialized string
representation:

```nim
import pkg/serde/json

type MyType {.serialize.} = object
  field1: string
  field2: bool

let mt = MyType(field1: "hw", field2: true)
assert mt.toJson == """{"field1":"hw","field2":true}"""
```

This comes in handy, for example, when sending API responses:

```nim
let availability = getAvailability(...)
return RestApiResponse.response(availability.toJson,
                                contentType="application/json")
```

## `std/json` drop-in replacment

`nim-serde` can be used as a drop-in replacement for the [standard library's `json`
module](https://nim-lang.org/docs/json.html), with a few notable improvements.

Instead of importing `std/json` into your application, `pkg/serde/json` can be imported
instead:

```diff
- import std/json
+ import pkg/serde/json
```

As with `std/json`, `%` can be used to serialize a type into a `JsonNode`:

```nim
import pkg/serde/json

assert %"hello" == newJString("hello")
```

And `%*` can be used to serialize objects:

```nim
import pkg/serde/json

let expected = newJObject()
expected["hello"] = newJString("world")
assert %*{"hello": "world"} == expected
```

As well, serialization of types can be overridden, and serialization of custom types can
be introduced. Here, we are overriding the serialization of `int`:

```nim
import pkg/serde/json

func `%`(i: int): JsonNode =
  newJInt(i + 1)

assert 1.toJson == "2"
```

## `parseJson` and exception tracking

Unfortunately, `std/json`'s `parseJson` can raise an `Exception`, so proper exception
tracking breaks, eg

```nim

## Fails to compile:
## Error: parseJson(me, false, false) can raise an unlisted exception: Exception

import std/json

{.push raises:[].}

type
  MyAppError = object of CatchableError

proc parseMe(me: string): JsonNode =
  try:
    return me.parseJson
  except CatchableError as error:
    raise newException(MyAppError, error.msg)

assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }
```

This is due to `std/json`'s `parseJson` incorrectly raising `Exception`. This can be
worked around by instead importing `serde` and calling its `JsonNode.parse` routine.
Note that `serde`'s `JsonNode.parse` returns a `Result[JsonNode, CatchableError]`
instead of just a plain `JsonNode` object as in `std/json`'s `parseJson`:

```nim
import pkg/serde/json

{.push raises:[].}

type
  MyAppError = object of CatchableError

proc parseMe(me: string): JsonNode {.raises: [MyAppError].} =
  without parsed =? JsonNode.parse(me), error:
    raise newException(MyAppError, error.msg)
  parsed

assert """{"hello":"world"}""".parseMe == %* { "hello": "world" }
```

## Known issues

There is a known issue when using mixins with generic overloaded procs like
`fromJson`. At the time of mixin call, only the `fromJson` overloads in scope of
the called mixin are available to be dispatched at runtime. There could be other
`fromJson` overloads declared in other modules, but are not in scope at the time
the mixin was called. Therefore, anytime `fromJson` is called targeting a
declared overload, it may or may not be dispatchable. This can be worked around
by forcing the `fromJson` overload into scope at compile time. For example, in
your application where the `fromJson` overload is defined, at the bottom of the
module add:
```nim
static: MyType.fromJson("")
```
This will ensure that the `MyType.fromJson` overload is dispatchable.

The basic types that serde supports should already have their overloads forced
in scope in [the `deserializer`
module](./serde/json/deserializer.nim#L340-L356).

For an illustration of the problem, please see this [narrow example](https://github.com/gmega/serialization-bug/tree/main/narrow) by [@gmega](https://github.com/gmega).