From a69828922311c218b81705b34711500d97d672e3 Mon Sep 17 00:00:00 2001 From: Felix Krause Date: Sat, 11 Nov 2023 15:36:10 +0100 Subject: [PATCH] New pragmas for customizing presentation * added yaml/style that defines representation pragmas `scalar` and `collection` * implemented those pragmas * updated docs. also fixes some invalid links in docs --- CHANGELOG.md | 3 ++ doc/api.txt | 14 ++--- doc/serialization.txt | 4 +- nimdoc.cfg | 1 + test/tnative.nim | 26 ++++++++++ yaml/annotations.nim | 3 ++ yaml/data.nim | 27 ++++++---- yaml/native.nim | 116 ++++++++++++++++++++++++++++++++---------- yaml/stream.nim | 6 +-- yaml/style.nim | 40 +++++++++++++++ 10 files changed, 190 insertions(+), 50 deletions(-) create mode 100644 yaml/style.nim diff --git a/CHANGELOG.md b/CHANGELOG.md index a30ba8f..b8e4886 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,9 @@ Features: + * New pragmas ``scalar`` and ``collection`` allow you to modify the + presentation style used for certain types and object fields. + Defined in new module ``yaml/style``. * The presenter now honors the node style set in the events it presents, if possible. So if a scalar is set to be a literal block scalar, it is presented as such unless impossible or presenter options specifically diff --git a/doc/api.txt b/doc/api.txt index c56a05e..73d8eae 100644 --- a/doc/api.txt +++ b/doc/api.txt @@ -26,7 +26,7 @@ Intermediate Representation The base of all YAML processing with NimYAML is the `YamlStream `_. This is basically an iterator over -`YamlStreamEvent `_ objects. Every proc that +`Event `_ objects. Every proc that represents a single stage of the loading or dumping process will either take a ``YamlStream`` as input or return a ``YamlStream``. Procs that implement the whole process in one step hide the ``YamlStream`` from the user. Every proc that @@ -45,7 +45,7 @@ Loading YAML ============ If you want to load YAML character data directly into a native Nim variable, you -can use `load `_. This is the easiest and +can use `load `_. This is the easiest and recommended way to load YAML data. This section gives an overview about how ``load`` is implemented. It is absolutely possible to reimplement the loading step using the low-level API. @@ -68,21 +68,21 @@ Dumping YAML ============ Dumping is preferredly done with -`dump `_, -which serializes a native Nim variable to a character stream. As with ``load``, +`dump `_, +which serializes a native Nim value to a character stream. As with ``load``, the following paragraph describes how ``dump`` is implemented using the low-level API. A Nim value is transformed into a ``YamlStream`` with -`represent `_. -Depending on the ``AnchorStyle`` you specify, this will transform ``ref`` +`represent `_. +Depending on the ``AnchorStyle`` you specify in the given ``SerializationOptions``, this will transform ``ref`` variables with multiple instances into anchored elements and aliases (for ``asTidy`` and ``asAlways``) or write the same element into all places it occurs (for ``asNone``). Be aware that if you use ``asNone``, the value you serialize might not round-trip. Transforming a ``YamlStream`` into YAML character data is done with -`present `_. +`present `_. You can choose from multiple presentation styles. ``psJson`` is not able to process some features of ``YamlStream`` s, the other styles support all features and are guaranteed to round-trip to the same ``YamlStream`` if you parse the diff --git a/doc/serialization.txt b/doc/serialization.txt index 8f6c8a7..c33ec4f 100644 --- a/doc/serialization.txt +++ b/doc/serialization.txt @@ -247,7 +247,7 @@ These tags are generated on the fly based on the types you instantiate ``Table`` or ``seq`` with. You may customize the tags used for your types by using the template -`setTagUri `_. It may not +`setTagUri `_. It may not be applied to scalar and collection types implemented by NimYAML, but you can for example use it on a certain ``seq`` type: @@ -386,7 +386,7 @@ serialization context via ``ctx.put``. Follow the following guidelines when implementing a custom ``representObject`` proc: - You can use the helper template - `presentTag `_ + `presentTag `_ for outputting the tag. - Always output the first token with a ``yAnchorNone``. Anchors will be set automatically by ``ref`` type handling. diff --git a/nimdoc.cfg b/nimdoc.cfg index 68fce6d..fdd4777 100644 --- a/nimdoc.cfg +++ b/nimdoc.cfg @@ -130,6 +130,7 @@ doc.file = """
  • yaml/parser
  • yaml/presenter
  • yaml/stream
    • +
  • yaml/style
  • yaml/taglib
  • yaml/tojson
    • diff --git a/test/tnative.nim b/test/tnative.nim index d3f976d..5595eab 100644 --- a/test/tnative.nim +++ b/test/tnative.nim @@ -66,6 +66,15 @@ type i*: int Child = object of Parent s*: string + + AsFlow {.collection: csFlow.} = object + a* {.scalar: ssSingleQuoted.}: string + b* {.scalar: ssDoubleQuoted.}: string + c* {.scalar: ssPlain.}: string + + OverrideColStyle = object + flowChild: AsFlow + blockChild {.collection: csBlock.}: AsFlow proc `$`(v: BetterInt): string {.borrow.} proc `==`(left, right: BetterInt): bool {.borrow.} @@ -661,3 +670,20 @@ suite "Serialization": "- !test:BetterInt 1\n" & "- !test:BetterInt 9_998_887\n" & "- !test:BetterInt 98_312\n", output + + test "Representation pragmas": + let input = OverrideColStyle( + flowChild: AsFlow(a: "abc", b: "abc", c: "abc"), + blockChild: AsFlow(a: "a\nc", b: "abc", c: "ab:") + ) + var output = blockOnlyDumper().dump(input) + assertStringEqual "flowChild: {\n" & + " a: 'abc',\n" & + " b: \"abc\",\n" & + " c: abc\n" & + " }\n" & + "blockChild:\n" & + " a: \"a\\\nc\"\n" & + " b: \"abc\"\n" & + " c: \"ab:\"\n", output + \ No newline at end of file diff --git a/yaml/annotations.nim b/yaml/annotations.nim index 639f221..bb9f336 100644 --- a/yaml/annotations.nim +++ b/yaml/annotations.nim @@ -11,6 +11,9 @@ ## This module provides annotations for object fields that customize ## (de)serialization behavior of those fields. +import style +export style + template defaultVal*(value : typed) {.pragma.} ## This annotation can be put on an object field. During deserialization, ## if no value for this field is given, the ``value`` parameter of this diff --git a/yaml/data.nim b/yaml/data.nim index 1ef69a9..806eb34 100644 --- a/yaml/data.nim +++ b/yaml/data.nim @@ -4,8 +4,17 @@ # See the file "copying.txt", included in this # distribution, for details about the copyright. +## ================ +## Module yaml/data +## ================ +## +## This module defines the event data structure which serves as internal +## representation of YAML structures. Every YAML document can be described +## as stream of events. + import hashes -import private/escaping +import style, private/escaping +export style type Anchor* = distinct string ## \ @@ -19,14 +28,6 @@ type Tag* = distinct string ## \ ## A ``Tag`` contains an URI, like for example ``"tag:yaml.org,2002:str"``. - ScalarStyle* = enum - ## Original style of the scalar (for input), - ## or desired style of the scalar (for output). - ssAny, ssPlain, ssSingleQuoted, ssDoubleQuoted, ssLiteral, ssFolded - - CollectionStyle* = enum - csAny, csBlock, csFlow, csPair - EventKind* = enum ## Kinds of YAML events that may occur in an ``YamlStream``. Event kinds ## are discussed in `YamlStreamEvent <#YamlStreamEvent>`_. @@ -303,8 +304,12 @@ proc `$`*(event: Event): string {.raises: [].} = of yamlEndDoc: result = "-DOC" if event.explicitDocumentEnd: result &= " ..." - of yamlStartMap: result = "+MAP" & renderAttrs(event.mapProperties) - of yamlStartSeq: result = "+SEQ" & renderAttrs(event.seqProperties) + of yamlStartMap: + result = "+MAP" & (if event.mapStyle == csFlow: " {}" else: "") & + renderAttrs(event.mapProperties) + of yamlStartSeq: + result = "+SEQ" & (if event.seqStyle == csFlow: " []" else: "") & + renderAttrs(event.seqProperties) of yamlScalar: result = "=VAL" & renderAttrs(event.scalarProperties, event.scalarStyle == ssPlain or diff --git a/yaml/native.nim b/yaml/native.nim index 7f3fafc..b8504ce 100644 --- a/yaml/native.nim +++ b/yaml/native.nim @@ -62,6 +62,8 @@ type nextAnchorId: string options*: SerializationOptions putImpl*: proc(ctx: var SerializationContext, e: Event) {.raises: [], closure.} + overridingScalarStyle*: ScalarStyle = ssAny + overridingCollectionStyle*: CollectionStyle = csAny ConstructionContext* = object ## Context information for the process of constructing Nim values from YAML. @@ -82,6 +84,30 @@ type proc put*(ctx: var SerializationContext, e: Event) {.raises: [].} = ctx.putImpl(ctx, e) +proc scalarStyleFor(ctx: var SerializationContext, t: typedesc): ScalarStyle = + if ctx.overridingScalarStyle != ssAny: + result = ctx.overridingScalarStyle + ctx.overridingScalarStyle = ssAny + else: + when compiles(t.hasCustomPragma(scalar)): + when t.hasCustomPragma(scalar): + result = t.getCustomPragmaVal(scalar) + else: result = ssAny + else: result = ssAny + ctx.overridingCollectionStyle = csAny + +proc collectionStyleFor(ctx: var SerializationContext, t: typedesc): CollectionStyle = + if ctx.overridingCollectionStyle != csAny: + result = ctx.overridingCollectionStyle + ctx.overridingCollectionStyle = csAny + else: + when compiles(t.hasCustomPragma(collection)): + when t.hasCustomPragma(collection): + result = t.getCustomPragmaVal(collection) + else: result = csAny + else: result = csAny + ctx.overridingScalarStyle = ssAny + # forward declares proc constructChild*[T]( @@ -175,7 +201,11 @@ proc safeTagUri(tag: Tag): string {.raises: [].} = except KeyError: internalError("Unexpected KeyError for Tag " & $tag) -proc newYamlConstructionError*(s: YamlStream, mark: Mark, msg: string): ref YamlConstructionError = +proc newYamlConstructionError*( + s: YamlStream, + mark: Mark, + msg: string, +): ref YamlConstructionError = result = newException(YamlConstructionError, msg) result.mark = mark if not s.getLastTokenContext(result.lineContent): @@ -226,7 +256,7 @@ proc representObject*( tag : Tag, ) {.raises: [].} = ## represents a string as YAML scalar - ctx.put(scalarEvent(value, tag, yAnchorNone)) + ctx.put(scalarEvent(value, tag, yAnchorNone, ctx.scalarStyleFor(string))) proc parseHex[T: int8|int16|int32|int64|uint8|uint16|uint32|uint64]( s: YamlStream, mark: Mark, val: string @@ -271,8 +301,10 @@ proc constructObject*[T: int8|int16|int32|int64]( ## constructs an integer value from a YAML scalar ctx.input.constructScalarItem(item, T): case item.numberStyle - of nsHex: result = parseHex[T](ctx.input, item.startPos, item.scalarContent) - of nsOctal: result = parseOctal[T](ctx.input, item.startPos, item.scalarContent) + of nsHex: + result = parseHex[T](ctx.input, item.startPos, item.scalarContent) + of nsOctal: + result = parseOctal[T](ctx.input, item.startPos, item.scalarContent) of nsDecimal: let nInt = parseBiggestInt(item.scalarContent) if nInt <= T.high: @@ -301,7 +333,7 @@ proc representObject*[T: int8|int16|int32|int64]( tag : Tag, ) {.raises: [].} = ## represents an integer value as YAML scalar - ctx.put(scalarEvent($value, tag, yAnchorNone)) + ctx.put(scalarEvent($value, tag, yAnchorNone, ctx.scalarStyleFor(T))) proc representObject*( ctx : var SerializationContext, @@ -312,7 +344,9 @@ proc representObject*( ## on 64-bit systems, this may cause a RangeDefect. # currently, sizeof(int) is at least sizeof(int32). - try: ctx.put(scalarEvent($int32(value), tag, yAnchorNone)) + try: + ctx.put(scalarEvent( + $int32(value), tag, yAnchorNone, ctx.scalarStyleFor(int))) except RangeDefect as rd: var e = newException(YamlSerializationError, rd.msg) e.parent = rd @@ -330,8 +364,10 @@ proc constructObject*[T: DefiniteUIntTypes]( ## construct an unsigned integer value from a YAML scalar ctx.input.constructScalarItem(item, T): case item.numberStyle - of nsHex: result = parseHex[T](ctx.input, item.startPos, item.scalarContent) - of nsOctal: result = parseOctal[T](ctx.input, item.startPos, item.scalarContent) + of nsHex: + result = parseHex[T](ctx.input, item.startPos, item.scalarContent) + of nsOctal: + result = parseOctal[T](ctx.input, item.startPos, item.scalarContent) else: let nUInt = parseBiggestUInt(item.scalarContent) if nUInt <= T.high: @@ -365,7 +401,7 @@ proc representObject*[T: uint8|uint16|uint32|uint64]( tag : Tag, ) {.raises: [].} = ## represents an unsigned integer value as YAML scalar - ctx.put(scalarEvent($value, tag, yAnchorNone)) + ctx.put(scalarEvent($value, tag, yAnchorNone, ctx.scalarStyleFor(T))) proc representObject*( ctx : var SerializationContext, @@ -374,7 +410,9 @@ proc representObject*( ) {.raises: [YamlSerializationError], inline.} = ## represent an unsigned integer of architecture-defined length by casting it ## to int32. on 64-bit systems, this may cause a RangeDefect. - try: ctx.put(scalarEvent($uint32(value), tag, yAnchorNone)) + try: + ctx.put(scalarEvent( + $uint32(value), tag, yAnchorNone, ctx.scalarStyleFor(uint))) except RangeDefect as rd: var e = newException(YamlSerializationError, rd.msg) e.parent = rd @@ -413,10 +451,10 @@ proc representObject*[T: float|float32|float64]( ) {.raises: [].} = ## represents a float value as YAML scalar case value - of Inf: ctx.put(scalarEvent(".inf", tag)) - of NegInf: ctx.put(scalarEvent("-.inf", tag)) - of NaN: ctx.put(scalarEvent(".nan", tag)) - else: ctx.put(scalarEvent($value, tag)) + of Inf: ctx.put(scalarEvent(".inf", tag, ctx.scalarStyleFor(T))) + of NegInf: ctx.put(scalarEvent("-.inf", tag, ctx.scalarStyleFor(T))) + of NaN: ctx.put(scalarEvent(".nan", tag, ctx.scalarStyleFor(T))) + else: ctx.put(scalarEvent($value, tag, ctx.scalarStyleFor(T))) proc yamlTag*(T: typedesc[bool]): Tag {.inline, raises: [].} = yTagBoolean @@ -441,7 +479,8 @@ proc representObject*( tag : Tag, ) {.raises: [].} = ## represents a bool value as a YAML scalar - ctx.put(scalarEvent(if value: "true" else: "false", tag, yAnchorNone)) + ctx.put(scalarEvent(if value: "true" else: "false", + tag, yAnchorNone, ctx.scalarStyleFor(bool))) proc constructObject*( ctx : var ConstructionContext, @@ -462,7 +501,7 @@ proc representObject*( tag : Tag ) {.raises: [].} = ## represents a char value as YAML scalar - ctx.put(scalarEvent("" & value, tag, yAnchorNone)) + ctx.put(scalarEvent("" & value, tag, yAnchorNone, ctx.scalarStyleFor(char))) proc yamlTag*(T: typedesc[Time]): Tag {.inline, raises: [].} = yTagTimestamp @@ -534,7 +573,8 @@ proc representObject*( tag : Tag, ) {.raises: [].} = let tmp = value.utc() - ctx.put(scalarEvent(tmp.format("yyyy-MM-dd'T'HH:mm:ss'Z'"))) + ctx.put(scalarEvent(tmp.format( + "yyyy-MM-dd'T'HH:mm:ss'Z'"), tag, yAnchorNone, ctx.scalarStyleFor(Time))) proc yamlTag*[I](T: typedesc[seq[I]]): Tag {.inline, raises: [].} = return nimTag("system:seq(" & safeTagUri(yamlTag(I)) & ')') @@ -578,7 +618,8 @@ proc representObject*[T]( tag : Tag, ) {.raises: [YamlSerializationError].} = ## represents a Nim seq as YAML sequence - ctx.put(startSeqEvent(tag = tag)) + ctx.put( + startSeqEvent(tag = tag, style = ctx.collectionStyleFor(type(value)))) for item in value: ctx.representChild(item) ctx.put(endSeqEvent()) @@ -610,7 +651,7 @@ proc representObject*[I, T]( tag : Tag, ) {.raises: [YamlSerializationError].} = ## represents a Nim array as YAML sequence - ctx.put(startSeqEvent(tag = tag)) + ctx.put(startSeqEvent(tag = tag, style = ctx.collectionStyleFor(array[I, T]))) for item in value: ctx.representChild(item) ctx.put(endSeqEvent()) @@ -646,7 +687,8 @@ proc representObject*[K, V]( tag : Tag, ) {.raises: [YamlSerializationError].} = ## represents a Nim Table as YAML mapping - ctx.put(startMapEvent(tag = tag)) + ctx.put( + startMapEvent(tag = tag, style = ctx.collectionStyleFor(Table[K, V]))) for key, value in value.pairs: ctx.representChild(key) ctx.representChild(value) @@ -694,7 +736,8 @@ proc representObject*[K, V]( value: OrderedTable[K, V], tag : Tag, ) {.raises: [YamlSerializationError].} = - ctx.put(startSeqEvent(tag = tag)) + ctx.put(startSeqEvent( + tag = tag, style = ctx.collectionStyleFor(OrderedTable[K, V]))) for key, value in value.pairs: ctx.put(startMapEvent()) ctx.representChild(key) @@ -1198,6 +1241,11 @@ proc recGenFieldRepresenters( `fieldName`, tag = if ctx.emitTag: yTagNimField else: yTagQuestionMark )) + when `fieldAccessor`.hasCustomPragma(scalar): + ctx.overridingScalarStyle = `fieldAccessor`.getCustomPragmaVal(scalar) + echo "set scalar style to ", $ctx.overridingScalarStyle + when `fieldAccessor`.hasCustomPragma(collection): + ctx.overridingCollectionStyle = `fieldAccessor`.getCustomPragmaVal(collection) ctx.representChild(`fieldAccessor`) ctx.put(endMapEvent()) ) @@ -1231,6 +1279,11 @@ proc recGenFieldRepresenters( `name`, tag = if ctx.emitTag: yTagNimField else: yTagQuestionMark )) + when `itemAccessor`.hasCustomPragma(scalar): + ctx.overridingScalarStyle = `itemAccessor`.getCustomPragmaVal(scalar) + echo "set scalar style to ", $ctx.overridingScalarStyle + when `itemAccessor`.hasCustomPragma(collection): + ctx.overridingCollectionStyle = `itemAccessor`.getCustomPragmaVal(collection) ctx.representChild(`itemAccessor`) ctx.put(endMapEvent()) ) @@ -1252,6 +1305,11 @@ proc recGenFieldRepresenters( if ctx.emitTag: yTagNimField else: yTagQuestionMark, yAnchorNone )) + when `childAccessor`.hasCustomPragma(scalar): + ctx.overridingScalarStyle = `childAccessor`.getCustomPragmaVal(scalar) + echo "set scalar style to ", $ctx.overridingScalarStyle + when `childAccessor`.hasCustomPragma(collection): + ctx.overridingCollectionStyle = `childAccessor`.getCustomPragmaVal(collection) ctx.representChild(`childAccessor`) when bool(`isVO`): ctx.put(endMapEvent()) when not `childAccessor`.hasCustomPragma(transient): @@ -1276,8 +1334,10 @@ proc representObject*[O: object]( tag : Tag, ) {.raises: [YamlSerializationError].} = ## represents a Nim object or tuple as YAML mapping - when isVariantObject(getType(O)): ctx.put(startSeqEvent(tag = tag)) - else: ctx.put(startMapEvent(tag = tag)) + when isVariantObject(getType(O)): + ctx.put(startSeqEvent(tag = tag, style = ctx.collectionStyleFor(O))) + else: + ctx.put(startMapEvent(tag = tag, style = ctx.collectionStyleFor(O))) genRepresentObject(O, value) when isVariantObject(getType(O)): ctx.put(endSeqEvent()) else: ctx.put(endMapEvent()) @@ -1288,7 +1348,7 @@ proc representObject*[O: tuple]( tag : Tag, ) {.raises: [YamlSerializationError].} = var fieldIndex = 0'i16 - ctx.put(startMapEvent(tag = tag)) + ctx.put(startMapEvent(tag = tag, style = ctx.collectionStyleFor(O))) for name, fvalue in fieldPairs(value): ctx.put(scalarEvent( name, @@ -1322,7 +1382,7 @@ proc representObject*[O: enum]( tag : Tag, ) {.raises: [].} = ## represents a Nim enum as YAML scalar - ctx.put(scalarEvent($value, tag, yAnchorNone)) + ctx.put(scalarEvent($value, tag, yAnchorNone, ctx.scalarStyleFor(O))) proc yamlTag*[O](T: typedesc[ref O]): Tag {.inline, raises: [].} = yamlTag(O) @@ -1621,7 +1681,7 @@ proc representChild*[O]( ctx : var SerializationContext, value: ref O, ) = - if isNil(value): ctx.put(scalarEvent("~", yTagNull)) + if isNil(value): ctx.put(scalarEvent("~", yTagNull, style = ctx.scalarStyleFor(O))) else: when nimvm: discard else: @@ -1643,6 +1703,8 @@ proc representChild*[O]( if not val.referenced: ctx.refs[p] = (val.a, true) ctx.put(aliasEvent(val.a)) + ctx.overridingScalarStyle = ssAny + ctx.overridingCollectionStyle = csAny return ctx.refs[p] = val nextAnchor(ctx.nextAnchorId, len(ctx.nextAnchorId) - 1) @@ -1675,7 +1737,7 @@ proc representChild*[T]( if value.isSome: ctx.representChild(value.get()) else: - ctx.put(scalarEvent("~", yTagNull)) + ctx.put(scalarEvent("~", yTagNull, style = ctx.scalarStyleFor(Option[T]))) proc representChild*[O]( ctx : var SerializationContext, diff --git a/yaml/stream.nim b/yaml/stream.nim index 2aa354e..77e5a72 100644 --- a/yaml/stream.nim +++ b/yaml/stream.nim @@ -8,9 +8,9 @@ ## Module yaml/stream ## ================== ## -## The stream API provides the basic data structure on which all low-level APIs -## operate. It is not named ``streams`` to not confuse it with the modle in the -## stdlib with that name. +## The stream API provides the basic data structure ``YamlStream`` on which all +## low-level APIs operate. It is not named ``streams`` to not confuse it with +## the module in the stdlib with that name. import data diff --git a/yaml/style.nim b/yaml/style.nim new file mode 100644 index 0000000..bc7dd0e --- /dev/null +++ b/yaml/style.nim @@ -0,0 +1,40 @@ +# NimYAML - YAML implementation in Nim +# (c) Copyright 2015-2023 Felix Krause +# +# See the file "copying.txt", included in this +# distribution, for details about the copyright. + +## ================= +## Module yaml/style +## ================= +## +## The style API provides enums describing the style of YAML nodes. +## It also provides custom pragmas with which you can define the style +## with which values should be serialized. + +type + ScalarStyle* = enum + ## Original style of the scalar (for input), + ## or desired style of the scalar (for output). + ssAny, ssPlain, ssSingleQuoted, ssDoubleQuoted, ssLiteral, ssFolded + + CollectionStyle* = enum + ## Original style of the collection (for input). + ## or desired style of the collection (for output). + csAny, csBlock, csFlow, csPair + +template scalar*(style: ScalarStyle) {.pragma.} + ## This annotation can be put on an object field or on a type. + ## It causes the value in the field or a value of this type + ## to be presented with the given scalar style if possible. + ## Ignored if the value does not serialize to a scalar. + ## + ## A pragma on a field overrides a pragma on the field's type. + +template collection*(style: CollectionStyle) {.pragma.} + ## This annotation can be put on an object field or on a type. + ## It causes the value in the field or a value of this type + ## to be presented with the given collection style if possible. + ## Ignored if the value does not serialize to a collection. + ## + ## A pragma on a field overrides a pragma on the field's type. \ No newline at end of file