nim-json-serialization/README.md

# nim-json-serialization

[![License: Apache](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
![Stability: experimental](https://img.shields.io/badge/stability-experimental-orange.svg)
![Github action](https://github.com/status-im/nim-json-serialization/actions/workflows/ci.yml/badge.svg)

Flexible JSON serialization does not rely on run-time type information.

## Overview
nim-json-serialization offers rich features on top of [nim-serialization](https://github.com/status-im/nim-serialization)
framework. The following is available but not an exhaustive list of features:

  - Decode into Nim data types efficiently without an intermediate token.
  - Able to parse full spec of JSON including the notorious JSON number.
  - Support stdlib/JsonNode out of the box.
  - While stdlib/JsonNode does not support the full spec of the Json number, we offer an alternative `JsonValueRef`.
  - Skipping Json value is an efficient process, no token is generated at all and at the same time, the grammar is checked.
    - Skipping is also free from custom serializer interference.
  - An entire Json value can be parsed into a valid Json document string. This string document can be parsed again without losing any information.
  - Custom serialization is easy and safe to implement with the help of many built-in parsers.
  - Nonstandard features are put behind flags. You can choose which features to switch on or off.
  - Because the intended usage of this library will be in a security-demanding application, we make sure malicious inputs will not crash
    this library through fuzz tests.
  - The user also can tweak certain limits of the lexer/parser behavior using the configuration object.
  - `createJsonFlavor` is a powerful way to prevent cross contamination between different subsystem using different custom serializar on the same type.

## Spec compliance
nim-json-serialization implements [RFC8259](https://datatracker.ietf.org/doc/html/rfc8259)
JSON spec and pass these test suites:

  - [JSONTestSuite](https://github.com/nst/JSONTestSuite)

## Switchable features
Many of these switchable features are widely used features in various projects but are not standard JSON features.
But you can access them using the flags:

  - **allowUnknownFields[=off]**: enable unknown fields to be skipped instead of throwing an error.
  - **requireAllFields[=off]**: if one of the required fields is missing, the serializer will throw an error.
  - **escapeHex[=off]**: JSON doesn't support `\xHH` escape sequence, but it is a common thing in many languages.
  - **relaxedEscape[=off]**: only '0x00'..'0x1F' can be prepended by escape char `\\`, turn this on and you can escape any char.
  - **portableInt[=off]**: set the limit of integer to `-2**53 + 1` and `+2**53 - 1`.
  - **trailingComma[=on]**: allow the presence of a trailing comma after the last object member or array element.
  - **allowComments[=on]**: JSOn standard doesn't mention about comments. Turn this on to parse both C style comments of `//..EOL` and `/* .. */`.
  - **leadingFraction[=on]**: something like `.123` is not a valid JSON number, but its widespread usage sometimes creeps into Json documents.
  - **integerPositiveSign[=on]**: `+123` is also not a valid JSON number, but since `-123` is a valid JSON number, why not parse it safely?

## Safety features
You can modify these default configurations to suit your needs.

  - **nestedDepthLimit: 512**: maximum depth of the nested structure, they are a combination of objects and arrays depth(0=disable).
  - **arrayElementsLimit: 0**: maximum number of allowed array elements(0=disable).
  - **objectMembersLimit: 0**: maximum number of key-value pairs in an object(0=disable).
  - **integerDigitsLimit: 128**: limit the maximum digits of the integer part of JSON number.
  - **fractionDigitsLimit: 128**: limit the maximum digits of faction part of JSON number.
  - **exponentDigitsLimit: 32**: limit the maximum digits of the exponent part of JSON number.
  - **stringLengthLimit: 0**: limit the maximum bytes of string(0=disable).

## Special types

  - **JsonString**: Use this type if you want to parse a Json value to a valid Json document contained in a string.
  - **JsonVoid**: Use this type to skip a valid Json value.
  - **JsonNumber**: Use this to parse a valid Json number including the fraction and exponent part.
    - Please note that this type is a generic, it support `uint64` and `string` as generic param.
    - The generic param will define the integer and exponent part as `uint64` or `string`.
    - If the generic param is `uint64`, overflow can happen, or max digit limit will apply.
    - If the generic param is `string`, the max digit limit will apply.
    - The fraction part is always a string to keep the leading zero of the fractional number.
  - **JsonValueRef**: Use this type to parse any valid Json value into something like stdlib/JsonNode.
    - `JsonValueRef` is using `JsonNumber` instead of `int` or `float` like stdlib/JsonNode.

## Flavor

While flags and limits are runtime configuration, flavor is a powerful compile time mechanism to prevent
cross contamination between different custom serializer operated the same type. For example,
`json-rpc` subsystem dan `json-rest` subsystem maybe have different custom serializer for the same `UInt256`.

Json-Flavor will make sure, the compiler picks the right serializer for the right subsystem.
You can use `useDefaultSerializationIn` to add serializers of a flavor to a specific type.

```Nim
# These are the parameters you can pass to `createJsonFlavor` to create a new flavor.

  FlavorName: untyped
  mimeTypeValue = "application/json"
  automaticObjectSerialization = false
  requireAllFields = true
  omitOptionalFields = true
  allowUnknownFields = true
  skipNullFields = false
```

```Nim
type
  OptionalFields = object
    one: Opt[string]
    two: Option[int]

createJsonFlavor OptJson
OptionalFields.useDefaultSerializationIn OptJson

`omitOptionalFields` is used by the Writer to ignore fields with null value.
`skipNullFields` is used by the Reader to ignore fields with null value.
```

## Decoder example
```nim
  type
    NimServer = object
      name: string
      port: int

    MixedServer = object
      name: JsonValueRef
      port: int

    StringServer = object
      name: JsonString
      port: JsonString

  # decode into native Nim
  var nim_native = Json.decode(rawJson, NimServer)

  # decode into mixed Nim + JsonValueRef
  var nim_mixed = Json.decode(rawJson, MixedServer)

  # decode any value into string
  var nim_string = Json.decode(rawJson, StringServer)

  # decode any valid JSON
  var json_value = Json.decode(rawJson, JsonValueRef)
```

## Load and save
```Nim
  var server = Json.loadFile("filename.json", Server)
  var server_string = Json.loadFile("filename.json", JsonString)

  Json.saveFile("filename.json", server)
```

## Objects
Decoding an object can be achieved via the `parseObject` template.
To parse the value, you can use one of the helper functions or use `readValue`.
`readObject` and `readObjectFields` iterators are also handy when creating a custom object parser.

```Nim
proc readValue*(r: var JsonReader, table: var Table[string, int]) =
  parseObject(r, key):
    table[key] = r.parseInt(int)
```

## Sets and list-like
Similar to `Object`, sets and list or array-like data structures can be parsed using
`parseArray` template. It comes in two variations, indexed and non-indexed.

Built-in `readValue` for regular `seq` and `array` is implemented for you.
No built-in `readValue` for `set` or `set-like` is provided, you must overload it yourself depending on your need.

```nim
type
  HoldArray = object
    data: array[3, int]

  HoldSeq = object
    data: seq[int]

  WelderFlag = enum
    TIG
    MIG
    MMA

  Welder = object
    flags: set[WelderFlag]

proc readValue*(r: var JsonReader, value: var HoldArray) =
  # parseArray with index, `i` can be any valid identifier
  r.parseArray(i):
    value.data[i] = r.parseInt(int)

proc readValue*(r: var JsonReader, value: var HoldSeq) =
  # parseArray without index
  r.parseArray:
    let lastPos = value.data.len
    value.data.setLen(lastPos + 1)
    readValue(r, value.data[lastPos])

proc readValue*(r: var JsonReader, value: var Welder) =
  # populating set also okay
  r.parseArray:
    value.flags.incl r.parseInt(int).WelderFlag
```

## Custom iterators
Using these custom iterators, you can have access to sub-token elements.

```Nim
customIntValueIt(r: var JsonReader; body: untyped)
customNumberValueIt(r: var JsonReader; body: untyped)
customStringValueIt(r: var JsonReader; limit: untyped; body: untyped)
customStringValueIt(r: var JsonReader; body: untyped)
```
## Convenience iterators

```Nim
readArray(r: var JsonReader, ElemType: typedesc): ElemType
readObjectFields(r: var JsonReader, KeyType: type): KeyType
readObjectFields(r: var JsonReader): string
readObject(r: var JsonReader, KeyType: type, ValueType: type): (KeyType, ValueType)
```

## Helper procs
When crafting a custom serializer, use these parsers, they are safe and intuitive.
Avoid using the lexer directly.

```Nim
tokKind(r: var JsonReader): JsonValueKind
parseString(r: var JsonReader, limit: int): string
parseString(r: var JsonReader): string
parseBool(r: var JsonReader): bool
parseNull(r: var JsonReader)
parseNumber(r: var JsonReader, T: type): JsonNumber[T: string or uint64]
parseNumber(r: var JsonReader, val: var JsonNumber)
toInt(r: var JsonReader, val: JsonNumber, T: type SomeInteger, portable: bool): T
parseInt(r: var JsonReader, T: type SomeInteger, portable: bool = false): T
toFloat(r: var JsonReader, val: JsonNumber, T: type SomeFloat): T
parseFloat(r: var JsonReader, T: type SomeFloat): T
parseAsString(r: var JsonReader, val: var string)
parseAsString(r: var JsonReader): JsonString
parseValue(r: var JsonReader, T: type): JsonValueRef[T: string or uint64]
parseValue(r: var JsonReader, val: var JsonValueRef)
parseArray(r: var JsonReader; body: untyped)
parseArray(r: var JsonReader; idx: untyped; body: untyped)
parseObject(r: var JsonReader, key: untyped, body: untyped)
parseObjectWithoutSkip(r: var JsonReader, key: untyped, body: untyped)
parseObjectSkipNullFields(r: var JsonReader, key: untyped, body: untyped)
parseObjectCustomKey(r: var JsonReader, keyAction: untyped, body: untyped)
parseJsonNode(r: var JsonReader): JsonNode
skipSingleJsValue(r: var JsonReader)
readRecordValue[T](r: var JsonReader, value: var T)
```

## Helper procs of JsonWriter

```Nim
beginRecord(w: var JsonWriter, T: type)
beginRecord(w: var JsonWriter)
endRecord(w: var JsonWriter)

writeObject(w: var JsonWriter, T: type)
writeObject(w: var JsonWriter)

writeFieldName(w: var JsonWriter, name: string)
writeField(w: var JsonWriter, name: string, value: auto)

iterator stepwiseArrayCreation[C](w: var JsonWriter, collection: C): auto
writeIterable(w: var JsonWriter, collection: auto)
writeArray[T](w: var JsonWriter, elements: openArray[T])

writeNumber[F,T](w: var JsonWriter[F], value: JsonNumber[T])
writeJsonValueRef[F,T](w: var JsonWriter[F], value: JsonValueRef[T])
```

## License

Licensed and distributed under either of

* MIT license: [LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT

or

* Apache License, Version 2.0, ([LICENSE-APACHEv2](LICENSE-APACHEv2) or http://www.apache.org/licenses/LICENSE-2.0)

at your option. These files may not be copied, modified, or distributed except according to those terms.