New pragmas for customizing presentation

* added yaml/style that defines representation pragmas
   `scalar` and `collection`
 * implemented those pragmas
 * updated docs. also fixes some invalid links in docs
This commit is contained in:
Felix Krause 2023-11-11 15:36:10 +01:00
parent 03c95da97c
commit a698289223
10 changed files with 190 additions and 50 deletions

View File

@ -2,6 +2,9 @@
Features:
* New pragmas ``scalar`` and ``collection`` allow you to modify the
presentation style used for certain types and object fields.
Defined in new module ``yaml/style``.
* The presenter now honors the node style set in the events it presents,
if possible. So if a scalar is set to be a literal block scalar, it is
presented as such unless impossible or presenter options specifically

View File

@ -26,7 +26,7 @@ Intermediate Representation
The base of all YAML processing with NimYAML is the
`YamlStream <api/stream.html#YamlStream>`_. This is basically an iterator over
`YamlStreamEvent <api/stream.html#YamlStreamEvent>`_ objects. Every proc that
`Event <api/data.html#Event>`_ objects. Every proc that
represents a single stage of the loading or dumping process will either take a
``YamlStream`` as input or return a ``YamlStream``. Procs that implement the
whole process in one step hide the ``YamlStream`` from the user. Every proc that
@ -45,7 +45,7 @@ Loading YAML
============
If you want to load YAML character data directly into a native Nim variable, you
can use `load <api/serialization.html#load,,K>`_. This is the easiest and
can use `load <api/loading.html#load%2C%2CK>`_. This is the easiest and
recommended way to load YAML data. This section gives an overview about how
``load`` is implemented. It is absolutely possible to reimplement the loading
step using the low-level API.
@ -68,21 +68,21 @@ Dumping YAML
============
Dumping is preferredly done with
`dump <api/serialization.html#dump,K,Stream,TagStyle,AnchorStyle,PresentationOptions>`_,
which serializes a native Nim variable to a character stream. As with ``load``,
`dump <api/dumping.html#dump%2CDumper%2CK>`_,
which serializes a native Nim value to a character stream. As with ``load``,
the following paragraph describes how ``dump`` is implemented using the
low-level API.
A Nim value is transformed into a ``YamlStream`` with
`represent <api/serialization.html#represent,T,TagStyle,AnchorStyle>`_.
Depending on the ``AnchorStyle`` you specify, this will transform ``ref``
`represent <api/native.html#represent%2CT%2CSerializationOptions>`_.
Depending on the ``AnchorStyle`` you specify in the given ``SerializationOptions``, this will transform ``ref``
variables with multiple instances into anchored elements and aliases (for
``asTidy`` and ``asAlways``) or write the same element into all places it
occurs (for ``asNone``). Be aware that if you use ``asNone``, the value you
serialize might not round-trip.
Transforming a ``YamlStream`` into YAML character data is done with
`present <api/presenter.html#present,YamlStream,Stream,TagLibrary,PresentationOptions>`_.
`present <api/presenter.html#present%2CYamlStream%2CStream%2CPresentationOptions>`_.
You can choose from multiple presentation styles. ``psJson`` is not able to
process some features of ``YamlStream`` s, the other styles support all features
and are guaranteed to round-trip to the same ``YamlStream`` if you parse the

View File

@ -247,7 +247,7 @@ These tags are generated on the fly based on the types you instantiate
``Table`` or ``seq`` with.
You may customize the tags used for your types by using the template
`setTagUri <yaml.taglib.html#setTagUri.t,typedesc,string>`_. It may not
`setTagUri <api/taglib.html#setTagUri.t%2Ctypedesc%2Cstring>`_. It may not
be applied to scalar and collection types implemented by NimYAML, but you can
for example use it on a certain ``seq`` type:
@ -386,7 +386,7 @@ serialization context via ``ctx.put``. Follow the following guidelines when
implementing a custom ``representObject`` proc:
- You can use the helper template
`presentTag <yaml.native.html#presentTag,SerializationContext,TagStyle>`_
`presentTag <api/native.html#presentTag%2CSerializationContext%2Ctypedesc>`_
for outputting the tag.
- Always output the first token with a ``yAnchorNone``. Anchors will be set
automatically by ``ref`` type handling.

View File

@ -130,6 +130,7 @@ doc.file = """
<li><a href="/api/parser.html">yaml/parser</a></li>
<li><a href="/api/presenter.html">yaml/presenter</a></li>
<li><a href="/api/stream.html">yaml/stream</a></li>
<li><a href="/api/style.html">yaml/style</a></li>
<li><a href="/api/taglib.html">yaml/taglib</a></li>
<li><a href="/api/tojson.html">yaml/tojson</a></li>
</ul>

View File

@ -67,6 +67,15 @@ type
Child = object of Parent
s*: string
AsFlow {.collection: csFlow.} = object
a* {.scalar: ssSingleQuoted.}: string
b* {.scalar: ssDoubleQuoted.}: string
c* {.scalar: ssPlain.}: string
OverrideColStyle = object
flowChild: AsFlow
blockChild {.collection: csBlock.}: AsFlow
proc `$`(v: BetterInt): string {.borrow.}
proc `==`(left, right: BetterInt): bool {.borrow.}
@ -661,3 +670,20 @@ suite "Serialization":
"- !test:BetterInt 1\n" &
"- !test:BetterInt 9_998_887\n" &
"- !test:BetterInt 98_312\n", output
test "Representation pragmas":
let input = OverrideColStyle(
flowChild: AsFlow(a: "abc", b: "abc", c: "abc"),
blockChild: AsFlow(a: "a\nc", b: "abc", c: "ab:")
)
var output = blockOnlyDumper().dump(input)
assertStringEqual "flowChild: {\n" &
" a: 'abc',\n" &
" b: \"abc\",\n" &
" c: abc\n" &
" }\n" &
"blockChild:\n" &
" a: \"a\\\nc\"\n" &
" b: \"abc\"\n" &
" c: \"ab:\"\n", output

View File

@ -11,6 +11,9 @@
## This module provides annotations for object fields that customize
## (de)serialization behavior of those fields.
import style
export style
template defaultVal*(value : typed) {.pragma.}
## This annotation can be put on an object field. During deserialization,
## if no value for this field is given, the ``value`` parameter of this

View File

@ -4,8 +4,17 @@
# See the file "copying.txt", included in this
# distribution, for details about the copyright.
## ================
## Module yaml/data
## ================
##
## This module defines the event data structure which serves as internal
## representation of YAML structures. Every YAML document can be described
## as stream of events.
import hashes
import private/escaping
import style, private/escaping
export style
type
Anchor* = distinct string ## \
@ -19,14 +28,6 @@ type
Tag* = distinct string ## \
## A ``Tag`` contains an URI, like for example ``"tag:yaml.org,2002:str"``.
ScalarStyle* = enum
## Original style of the scalar (for input),
## or desired style of the scalar (for output).
ssAny, ssPlain, ssSingleQuoted, ssDoubleQuoted, ssLiteral, ssFolded
CollectionStyle* = enum
csAny, csBlock, csFlow, csPair
EventKind* = enum
## Kinds of YAML events that may occur in an ``YamlStream``. Event kinds
## are discussed in `YamlStreamEvent <#YamlStreamEvent>`_.
@ -303,8 +304,12 @@ proc `$`*(event: Event): string {.raises: [].} =
of yamlEndDoc:
result = "-DOC"
if event.explicitDocumentEnd: result &= " ..."
of yamlStartMap: result = "+MAP" & renderAttrs(event.mapProperties)
of yamlStartSeq: result = "+SEQ" & renderAttrs(event.seqProperties)
of yamlStartMap:
result = "+MAP" & (if event.mapStyle == csFlow: " {}" else: "") &
renderAttrs(event.mapProperties)
of yamlStartSeq:
result = "+SEQ" & (if event.seqStyle == csFlow: " []" else: "") &
renderAttrs(event.seqProperties)
of yamlScalar:
result = "=VAL" & renderAttrs(event.scalarProperties,
event.scalarStyle == ssPlain or

View File

@ -62,6 +62,8 @@ type
nextAnchorId: string
options*: SerializationOptions
putImpl*: proc(ctx: var SerializationContext, e: Event) {.raises: [], closure.}
overridingScalarStyle*: ScalarStyle = ssAny
overridingCollectionStyle*: CollectionStyle = csAny
ConstructionContext* = object
## Context information for the process of constructing Nim values from YAML.
@ -82,6 +84,30 @@ type
proc put*(ctx: var SerializationContext, e: Event) {.raises: [].} =
ctx.putImpl(ctx, e)
proc scalarStyleFor(ctx: var SerializationContext, t: typedesc): ScalarStyle =
if ctx.overridingScalarStyle != ssAny:
result = ctx.overridingScalarStyle
ctx.overridingScalarStyle = ssAny
else:
when compiles(t.hasCustomPragma(scalar)):
when t.hasCustomPragma(scalar):
result = t.getCustomPragmaVal(scalar)
else: result = ssAny
else: result = ssAny
ctx.overridingCollectionStyle = csAny
proc collectionStyleFor(ctx: var SerializationContext, t: typedesc): CollectionStyle =
if ctx.overridingCollectionStyle != csAny:
result = ctx.overridingCollectionStyle
ctx.overridingCollectionStyle = csAny
else:
when compiles(t.hasCustomPragma(collection)):
when t.hasCustomPragma(collection):
result = t.getCustomPragmaVal(collection)
else: result = csAny
else: result = csAny
ctx.overridingScalarStyle = ssAny
# forward declares
proc constructChild*[T](
@ -175,7 +201,11 @@ proc safeTagUri(tag: Tag): string {.raises: [].} =
except KeyError:
internalError("Unexpected KeyError for Tag " & $tag)
proc newYamlConstructionError*(s: YamlStream, mark: Mark, msg: string): ref YamlConstructionError =
proc newYamlConstructionError*(
s: YamlStream,
mark: Mark,
msg: string,
): ref YamlConstructionError =
result = newException(YamlConstructionError, msg)
result.mark = mark
if not s.getLastTokenContext(result.lineContent):
@ -226,7 +256,7 @@ proc representObject*(
tag : Tag,
) {.raises: [].} =
## represents a string as YAML scalar
ctx.put(scalarEvent(value, tag, yAnchorNone))
ctx.put(scalarEvent(value, tag, yAnchorNone, ctx.scalarStyleFor(string)))
proc parseHex[T: int8|int16|int32|int64|uint8|uint16|uint32|uint64](
s: YamlStream, mark: Mark, val: string
@ -271,8 +301,10 @@ proc constructObject*[T: int8|int16|int32|int64](
## constructs an integer value from a YAML scalar
ctx.input.constructScalarItem(item, T):
case item.numberStyle
of nsHex: result = parseHex[T](ctx.input, item.startPos, item.scalarContent)
of nsOctal: result = parseOctal[T](ctx.input, item.startPos, item.scalarContent)
of nsHex:
result = parseHex[T](ctx.input, item.startPos, item.scalarContent)
of nsOctal:
result = parseOctal[T](ctx.input, item.startPos, item.scalarContent)
of nsDecimal:
let nInt = parseBiggestInt(item.scalarContent)
if nInt <= T.high:
@ -301,7 +333,7 @@ proc representObject*[T: int8|int16|int32|int64](
tag : Tag,
) {.raises: [].} =
## represents an integer value as YAML scalar
ctx.put(scalarEvent($value, tag, yAnchorNone))
ctx.put(scalarEvent($value, tag, yAnchorNone, ctx.scalarStyleFor(T)))
proc representObject*(
ctx : var SerializationContext,
@ -312,7 +344,9 @@ proc representObject*(
## on 64-bit systems, this may cause a RangeDefect.
# currently, sizeof(int) is at least sizeof(int32).
try: ctx.put(scalarEvent($int32(value), tag, yAnchorNone))
try:
ctx.put(scalarEvent(
$int32(value), tag, yAnchorNone, ctx.scalarStyleFor(int)))
except RangeDefect as rd:
var e = newException(YamlSerializationError, rd.msg)
e.parent = rd
@ -330,8 +364,10 @@ proc constructObject*[T: DefiniteUIntTypes](
## construct an unsigned integer value from a YAML scalar
ctx.input.constructScalarItem(item, T):
case item.numberStyle
of nsHex: result = parseHex[T](ctx.input, item.startPos, item.scalarContent)
of nsOctal: result = parseOctal[T](ctx.input, item.startPos, item.scalarContent)
of nsHex:
result = parseHex[T](ctx.input, item.startPos, item.scalarContent)
of nsOctal:
result = parseOctal[T](ctx.input, item.startPos, item.scalarContent)
else:
let nUInt = parseBiggestUInt(item.scalarContent)
if nUInt <= T.high:
@ -365,7 +401,7 @@ proc representObject*[T: uint8|uint16|uint32|uint64](
tag : Tag,
) {.raises: [].} =
## represents an unsigned integer value as YAML scalar
ctx.put(scalarEvent($value, tag, yAnchorNone))
ctx.put(scalarEvent($value, tag, yAnchorNone, ctx.scalarStyleFor(T)))
proc representObject*(
ctx : var SerializationContext,
@ -374,7 +410,9 @@ proc representObject*(
) {.raises: [YamlSerializationError], inline.} =
## represent an unsigned integer of architecture-defined length by casting it
## to int32. on 64-bit systems, this may cause a RangeDefect.
try: ctx.put(scalarEvent($uint32(value), tag, yAnchorNone))
try:
ctx.put(scalarEvent(
$uint32(value), tag, yAnchorNone, ctx.scalarStyleFor(uint)))
except RangeDefect as rd:
var e = newException(YamlSerializationError, rd.msg)
e.parent = rd
@ -413,10 +451,10 @@ proc representObject*[T: float|float32|float64](
) {.raises: [].} =
## represents a float value as YAML scalar
case value
of Inf: ctx.put(scalarEvent(".inf", tag))
of NegInf: ctx.put(scalarEvent("-.inf", tag))
of NaN: ctx.put(scalarEvent(".nan", tag))
else: ctx.put(scalarEvent($value, tag))
of Inf: ctx.put(scalarEvent(".inf", tag, ctx.scalarStyleFor(T)))
of NegInf: ctx.put(scalarEvent("-.inf", tag, ctx.scalarStyleFor(T)))
of NaN: ctx.put(scalarEvent(".nan", tag, ctx.scalarStyleFor(T)))
else: ctx.put(scalarEvent($value, tag, ctx.scalarStyleFor(T)))
proc yamlTag*(T: typedesc[bool]): Tag {.inline, raises: [].} = yTagBoolean
@ -441,7 +479,8 @@ proc representObject*(
tag : Tag,
) {.raises: [].} =
## represents a bool value as a YAML scalar
ctx.put(scalarEvent(if value: "true" else: "false", tag, yAnchorNone))
ctx.put(scalarEvent(if value: "true" else: "false",
tag, yAnchorNone, ctx.scalarStyleFor(bool)))
proc constructObject*(
ctx : var ConstructionContext,
@ -462,7 +501,7 @@ proc representObject*(
tag : Tag
) {.raises: [].} =
## represents a char value as YAML scalar
ctx.put(scalarEvent("" & value, tag, yAnchorNone))
ctx.put(scalarEvent("" & value, tag, yAnchorNone, ctx.scalarStyleFor(char)))
proc yamlTag*(T: typedesc[Time]): Tag {.inline, raises: [].} = yTagTimestamp
@ -534,7 +573,8 @@ proc representObject*(
tag : Tag,
) {.raises: [].} =
let tmp = value.utc()
ctx.put(scalarEvent(tmp.format("yyyy-MM-dd'T'HH:mm:ss'Z'")))
ctx.put(scalarEvent(tmp.format(
"yyyy-MM-dd'T'HH:mm:ss'Z'"), tag, yAnchorNone, ctx.scalarStyleFor(Time)))
proc yamlTag*[I](T: typedesc[seq[I]]): Tag {.inline, raises: [].} =
return nimTag("system:seq(" & safeTagUri(yamlTag(I)) & ')')
@ -578,7 +618,8 @@ proc representObject*[T](
tag : Tag,
) {.raises: [YamlSerializationError].} =
## represents a Nim seq as YAML sequence
ctx.put(startSeqEvent(tag = tag))
ctx.put(
startSeqEvent(tag = tag, style = ctx.collectionStyleFor(type(value))))
for item in value: ctx.representChild(item)
ctx.put(endSeqEvent())
@ -610,7 +651,7 @@ proc representObject*[I, T](
tag : Tag,
) {.raises: [YamlSerializationError].} =
## represents a Nim array as YAML sequence
ctx.put(startSeqEvent(tag = tag))
ctx.put(startSeqEvent(tag = tag, style = ctx.collectionStyleFor(array[I, T])))
for item in value: ctx.representChild(item)
ctx.put(endSeqEvent())
@ -646,7 +687,8 @@ proc representObject*[K, V](
tag : Tag,
) {.raises: [YamlSerializationError].} =
## represents a Nim Table as YAML mapping
ctx.put(startMapEvent(tag = tag))
ctx.put(
startMapEvent(tag = tag, style = ctx.collectionStyleFor(Table[K, V])))
for key, value in value.pairs:
ctx.representChild(key)
ctx.representChild(value)
@ -694,7 +736,8 @@ proc representObject*[K, V](
value: OrderedTable[K, V],
tag : Tag,
) {.raises: [YamlSerializationError].} =
ctx.put(startSeqEvent(tag = tag))
ctx.put(startSeqEvent(
tag = tag, style = ctx.collectionStyleFor(OrderedTable[K, V])))
for key, value in value.pairs:
ctx.put(startMapEvent())
ctx.representChild(key)
@ -1198,6 +1241,11 @@ proc recGenFieldRepresenters(
`fieldName`,
tag = if ctx.emitTag: yTagNimField else: yTagQuestionMark
))
when `fieldAccessor`.hasCustomPragma(scalar):
ctx.overridingScalarStyle = `fieldAccessor`.getCustomPragmaVal(scalar)
echo "set scalar style to ", $ctx.overridingScalarStyle
when `fieldAccessor`.hasCustomPragma(collection):
ctx.overridingCollectionStyle = `fieldAccessor`.getCustomPragmaVal(collection)
ctx.representChild(`fieldAccessor`)
ctx.put(endMapEvent())
)
@ -1231,6 +1279,11 @@ proc recGenFieldRepresenters(
`name`,
tag = if ctx.emitTag: yTagNimField else: yTagQuestionMark
))
when `itemAccessor`.hasCustomPragma(scalar):
ctx.overridingScalarStyle = `itemAccessor`.getCustomPragmaVal(scalar)
echo "set scalar style to ", $ctx.overridingScalarStyle
when `itemAccessor`.hasCustomPragma(collection):
ctx.overridingCollectionStyle = `itemAccessor`.getCustomPragmaVal(collection)
ctx.representChild(`itemAccessor`)
ctx.put(endMapEvent())
)
@ -1252,6 +1305,11 @@ proc recGenFieldRepresenters(
if ctx.emitTag: yTagNimField else: yTagQuestionMark,
yAnchorNone
))
when `childAccessor`.hasCustomPragma(scalar):
ctx.overridingScalarStyle = `childAccessor`.getCustomPragmaVal(scalar)
echo "set scalar style to ", $ctx.overridingScalarStyle
when `childAccessor`.hasCustomPragma(collection):
ctx.overridingCollectionStyle = `childAccessor`.getCustomPragmaVal(collection)
ctx.representChild(`childAccessor`)
when bool(`isVO`): ctx.put(endMapEvent())
when not `childAccessor`.hasCustomPragma(transient):
@ -1276,8 +1334,10 @@ proc representObject*[O: object](
tag : Tag,
) {.raises: [YamlSerializationError].} =
## represents a Nim object or tuple as YAML mapping
when isVariantObject(getType(O)): ctx.put(startSeqEvent(tag = tag))
else: ctx.put(startMapEvent(tag = tag))
when isVariantObject(getType(O)):
ctx.put(startSeqEvent(tag = tag, style = ctx.collectionStyleFor(O)))
else:
ctx.put(startMapEvent(tag = tag, style = ctx.collectionStyleFor(O)))
genRepresentObject(O, value)
when isVariantObject(getType(O)): ctx.put(endSeqEvent())
else: ctx.put(endMapEvent())
@ -1288,7 +1348,7 @@ proc representObject*[O: tuple](
tag : Tag,
) {.raises: [YamlSerializationError].} =
var fieldIndex = 0'i16
ctx.put(startMapEvent(tag = tag))
ctx.put(startMapEvent(tag = tag, style = ctx.collectionStyleFor(O)))
for name, fvalue in fieldPairs(value):
ctx.put(scalarEvent(
name,
@ -1322,7 +1382,7 @@ proc representObject*[O: enum](
tag : Tag,
) {.raises: [].} =
## represents a Nim enum as YAML scalar
ctx.put(scalarEvent($value, tag, yAnchorNone))
ctx.put(scalarEvent($value, tag, yAnchorNone, ctx.scalarStyleFor(O)))
proc yamlTag*[O](T: typedesc[ref O]): Tag {.inline, raises: [].} = yamlTag(O)
@ -1621,7 +1681,7 @@ proc representChild*[O](
ctx : var SerializationContext,
value: ref O,
) =
if isNil(value): ctx.put(scalarEvent("~", yTagNull))
if isNil(value): ctx.put(scalarEvent("~", yTagNull, style = ctx.scalarStyleFor(O)))
else:
when nimvm: discard
else:
@ -1643,6 +1703,8 @@ proc representChild*[O](
if not val.referenced:
ctx.refs[p] = (val.a, true)
ctx.put(aliasEvent(val.a))
ctx.overridingScalarStyle = ssAny
ctx.overridingCollectionStyle = csAny
return
ctx.refs[p] = val
nextAnchor(ctx.nextAnchorId, len(ctx.nextAnchorId) - 1)
@ -1675,7 +1737,7 @@ proc representChild*[T](
if value.isSome:
ctx.representChild(value.get())
else:
ctx.put(scalarEvent("~", yTagNull))
ctx.put(scalarEvent("~", yTagNull, style = ctx.scalarStyleFor(Option[T])))
proc representChild*[O](
ctx : var SerializationContext,

View File

@ -8,9 +8,9 @@
## Module yaml/stream
## ==================
##
## The stream API provides the basic data structure on which all low-level APIs
## operate. It is not named ``streams`` to not confuse it with the modle in the
## stdlib with that name.
## The stream API provides the basic data structure ``YamlStream`` on which all
## low-level APIs operate. It is not named ``streams`` to not confuse it with
## the module in the stdlib with that name.
import data

40
yaml/style.nim Normal file
View File

@ -0,0 +1,40 @@
# NimYAML - YAML implementation in Nim
# (c) Copyright 2015-2023 Felix Krause
#
# See the file "copying.txt", included in this
# distribution, for details about the copyright.
## =================
## Module yaml/style
## =================
##
## The style API provides enums describing the style of YAML nodes.
## It also provides custom pragmas with which you can define the style
## with which values should be serialized.
type
ScalarStyle* = enum
## Original style of the scalar (for input),
## or desired style of the scalar (for output).
ssAny, ssPlain, ssSingleQuoted, ssDoubleQuoted, ssLiteral, ssFolded
CollectionStyle* = enum
## Original style of the collection (for input).
## or desired style of the collection (for output).
csAny, csBlock, csFlow, csPair
template scalar*(style: ScalarStyle) {.pragma.}
## This annotation can be put on an object field or on a type.
## It causes the value in the field or a value of this type
## to be presented with the given scalar style if possible.
## Ignored if the value does not serialize to a scalar.
##
## A pragma on a field overrides a pragma on the field's type.
template collection*(style: CollectionStyle) {.pragma.}
## This annotation can be put on an object field or on a type.
## It causes the value in the field or a value of this type
## to be presented with the given collection style if possible.
## Ignored if the value does not serialize to a collection.
##
## A pragma on a field overrides a pragma on the field's type.