From 0a7f87a539a816dd28ed1b4a9d9a98cdb1dfc056 Mon Sep 17 00:00:00 2001 From: Felix Krause Date: Mon, 15 Feb 2016 22:54:05 +0100 Subject: [PATCH] Improved docs. Fixed serialization problem. * Added serialization.txt to doc * Fixed rendering problem in api.txt * Use explicit tag when ref type renders to a scalar that can be parsed to !!null * Added test cases for this ref type fix --- config.nims | 1 + doc/api.txt | 20 +++--- doc/serialization.txt | 139 ++++++++++++++++++++++++++++++++++++++ doc/testing.html | 3 +- nimdoc.cfg | 3 +- private/serialization.nim | 10 ++- test/serializing.nim | 30 ++++++++ 7 files changed, 192 insertions(+), 14 deletions(-) create mode 100644 doc/serialization.txt diff --git a/config.nims b/config.nims index 0d4c5a9..4d6251c 100644 --- a/config.nims +++ b/config.nims @@ -23,6 +23,7 @@ task documentation, "Generate documentation": exec r"nim doc2 -o:docout/yaml.html --docSeeSrcUrl:https://github.com/flyx/NimYAML/blob/`git log -n 1 --format=%H` yaml" exec r"nim rst2html -o:docout/index.html doc/index.txt" exec r"nim rst2html -o:docout/api.html doc/api.txt" + exec r"nim rst2html -o:docout/serialization.html doc/serialization.txt" exec "cp doc/docutils.css doc/style.css doc/testing.html doc/processing.svg docout" setCommand "nop" diff --git a/doc/api.txt b/doc/api.txt index 2f6cc9f..296a0d9 100644 --- a/doc/api.txt +++ b/doc/api.txt @@ -1,9 +1,9 @@ -=== -API -=== +============ +API Overview +============ -Overview -======== +Introduction +============ NimYAML advocates parsing YAML input into native Nim types. Basic Nim library types like integers, floats and strings, as well as all tuples, enums and @@ -51,10 +51,10 @@ way to load YAML data. The following paragraphs will explain the steps involved. For parsing, a `YamlParser `_ object is needed. This object stores some state while parsing that may be useful for error reporting to the user. The -`parse `_ -proc implements the YAML processing step of the same name. All syntax errors in -the input character stream are processed by ``parse``, which will raise a -``YamlParserError`` if it encounters a syntax error. +`parse `_ proc implements the YAML processing +step of the same name. All syntax errors in the input character stream are +processed by ``parse``, which will raise a ``YamlParserError`` if it encounters +a syntax error. Transforming a ``YamlStream`` to a native YAML object is done via ``construct``. It skips the ``compose`` step for efficiency reasons. As Nim is @@ -81,6 +81,6 @@ that if you use ``asNone``, the value you serialize might not round-trip. Transforming a ``YamlStream`` into YAML character data is done with `present `_. You can choose from multiple presentation styles. ``psJson`` is not able to -process some features of ``YamlStream``s, the other styles support all features +process some features of ``YamlStream`` s, the other styles support all features and are guaranteed to round-trip to the same ``YamlStream`` if you parse the generated YAML character stream again. diff --git a/doc/serialization.txt b/doc/serialization.txt new file mode 100644 index 0000000..4708f79 --- /dev/null +++ b/doc/serialization.txt @@ -0,0 +1,139 @@ +====================== +Serialization Overview +====================== + +Introduction +============ + +NimYAML tries hard to make transforming YAML characters streams to native Nim +types and vice versa as easy as possible. In simple scenarios, you might not +need anything else than the two procs +`dump `_ and +`load `_. On the other side, the process should be as +customizable as possible to allow the user to tightly control how the generated +YAML character stream will look and how a YAML character stream is interpreted. + +An important thing to remember in NimYAML is that unlike in interpreted +languages like Ruby, Nim cannot load a YAML character stream without knowing the +resulting type beforehand. For example, if you want to load this piece of YAML: + +.. code-block:: yaml + %YAML 1.2 + --- !nim:system:seq(nim:system:int8) + - 1 + - 2 + - 3 + +You would need to know that it will load a ``seq[int8]`` *at compile time*. This +is not really a problem because without knowing which type you will load, you +cannot do anything useful with the result afterwards in the code. But it may be +unfamiliar for programmers who are used to the YAML libraries of Python or Ruby. + +Supported Types +=============== + +NimYAML supports a growing number of types of Nim's ``system`` module and +standard library, and it also supports user-defined object, tuple and enum types +out of the box. + +**Important**: NimYAML currently does not support polymorphism or variant +object types. This may be added in the future. + +This also means that NimYAML is generally able to work with object, tuple and +enum types defined in the standard library or a third-party library without +further configuration. + +Scalar Types +------------ + +The following integer types are supported by NimYAML: ``int8``, ``int16``, +``int32``, ``int64``, ``uint8``, ``uint16``, ``uint32``, ``uint64``. Note that +the ``int`` type is *not* supported, since its sice varies based on the target +operation system. This makes it unfit for usage within NimYAML, because a value +might not round-trip between a binary for a 32-bit OS and another binary for a +64-bit OS. The same goes for ``uint``. + +The floating point types ``float32`` and ``float64`` are also supported, but +``float`` is not, for the same reason. + +``string`` is supported and one of the few Nim types which directly map to a +standard YAML type. ``char`` is also supported. + +To support new scalar types, you must implement the ``constructObject()`` and +``representObject()`` procs on that type (see below). + +Collection Types +---------------- + +NimYAML supports Nim's ``seq`` and ``Table`` types out of the box. Unlike the +native YAML types ``!!seq`` and ``!!map``, ``seq`` and ``Table`` define the type +of all their items (or keys and values). So YAML objects with heterogeneous +types in them cannot be loaded to Nim collection types. For example, this +sequence: + +.. code-block:: yaml + %YAML 1.2 + --- !!seq + - !!int 1 + - !!string foo + +Cannot be loaded to a Nim ``seq``. For this reason, you cannot load YAML's +native ``!!map`` and ``!!seq`` types directly into Nim types. + +Reference Types +--------------- + +A reference to any supported non-reference type (including user defined types, +see below) is supported by NimYAML. A reference type will be treated like its +base type, but NimYAML is able to detect multiple references to the same object +and dump the structure properly with anchors and aliases in place. It is +possible to dump and load cyclic data structures without further configuration. +It is possible for reference types to hold a ``nil`` value, which will be mapped +to the ``!!null`` YAML scalar type. + +Pointer types are not supported because it seems dangerous to automatically +allocate memory which the user must then manually deallocate. + +User Defined Types +------------------ + +For an object or tuple type to be directly usable with NimYAML, the following +conditions must be met: + +- Every type contained in the object/tuple must be supported +- All fields of an object type must be accessible from the code position where + you call NimYAML. If an object has non-public member fields, it can only be + processed in the module where it is defined. +- The object may not contain a ``case`` clause. +- The object may not have a generic parameter + +NimYAML will present enum types as YAML scalars, and tuple and object types as +YAML maps. Some of the conditions above may be loosened in future releases. + +Tags +==== + +NimYAML uses local tags to represent Nim types that do not map directly to a +YAML type. For example, ``int8`` is presented with the tag ``!nim:system:int8``. +Tags are mostly unnecessary when loading YAML data because the user already +defines the target Nim type which usually defines all types of the structure. +However, there is one case where a tag is necessary: A reference type with the +value ``nil`` is represented in YAML as a ``!!null`` scalar. This will be +automatically detected by type guessing, but if it is for example a reference to +a string with the value ``"~"``, it must be tagged with ``!!string``, because +otherwise, it would be loaded as ``nil``. + +As you might have noticed in the example above, the YAML tag of a ``seq`` +depends on its generic type parameter. The same applies to ``Table``. So, a +table that maps ``int8`` to string sequences would be presented with the tag +``!nim:tables:Table(nim:system:int8,nim:system:seq(tag:yaml.org,2002:string))``. +These tags are generated on the fly based on the types you instantiate +``Table`` or ``seq`` with. + +You may customize the tags used for your types by using the template +`setTagUriForType `_. It may not +be applied to scalar and collection types implemented by NimYAML, but you can +for example use it on a certain ``seq`` type: + +.. code-block:: nim + setTagUriForType(seq[string], "!nim:my:seq") diff --git a/doc/testing.html b/doc/testing.html index 780112f..7175075 100644 --- a/doc/testing.html +++ b/doc/testing.html @@ -15,8 +15,9 @@ NimYAML Home Testing Ground - API: + Docs: Overview + Serialization Module yaml
diff --git a/nimdoc.cfg b/nimdoc.cfg index 4d4c787..bfda4d2 100644 --- a/nimdoc.cfg +++ b/nimdoc.cfg @@ -107,8 +107,9 @@ doc.file = """ NimYAML Home Testing Ground - API: + Docs: Overview + Serialization Module yaml
diff --git a/private/serialization.nim b/private/serialization.nim index 1eb57bf..180d26e 100644 --- a/private/serialization.nim +++ b/private/serialization.nim @@ -691,19 +691,25 @@ proc representObject*[O](value: ref O, ts: TagStyle, c: SerializationContext): cast[AnchorId](p)) return c.refsList.add(initRefNodeData(p)) - let a = if c.style == asAlways: AnchorId(c.refsList.high) else: + let + a = if c.style == asAlways: AnchorId(c.refsList.high) else: cast[AnchorId](p) + childTagStyle = if ts == tsAll: tsAll else: tsRootOnly result = iterator(): YamlStreamEvent = - var child = representObject(value[], ts, c) + var child = representObject(value[], childTagStyle, c) var first = child() assert(not finished(child)) case first.kind of yamlStartMap: first.mapAnchor = a + if ts == tsNone: first.mapTag = yTagQuestionMark of yamlStartSequence: first.seqAnchor = a + if ts == tsNone: first.seqTag = yTagQuestionMark of yamlScalar: first.scalarAnchor = a + if ts == tsNone and guessType(first.scalarContent) != yTypeNull: + first.scalarTag = yTagQuestionMark else: discard yield first while true: diff --git a/test/serializing.nim b/test/serializing.nim index 9307c9f..072733a 100644 --- a/test/serializing.nim +++ b/test/serializing.nim @@ -234,6 +234,7 @@ next: echo "line ", parser.getLineNumber, ", column ", parser.getColNumber, ": ", ex.msg echo parser.getLineContent + raise ex assert(result.len == 3) assert(result[0].value == "a") @@ -242,4 +243,33 @@ next: assert(result[0].next == result[1]) assert(result[1].next == result[2]) assert(result[2].next == result[0]) + + test "Serialization: Load nil values": + let input = newStringStream("- ~\n- !!str ~") + var + result: seq[ref string] + parser = newYamlParser(tagLib) + events = parser.parse(input) + try: + construct(events, result) + except YamlConstructionError: + let ex = (ref YamlConstructionError)(getCurrentException()) + echo "line ", parser.getLineNumber, ", column ", + parser.getColNumber, ": ", ex.msg + echo parser.getLineContent + raise ex + + assert(result.len == 2) + assert(result[0] == nil) + assert(result[1][] == "~") + + test "Serialization: Represent nil values": + var input = newSeq[ref string]() + input.add(nil) + input.add(new string) + input[1][] = "~" + var output = newStringStream() + dump(input, output, psBlockOnly, tsRootOnly) + assertStringEqual "%YAML 1.2\n--- !nim:system:seq(tag:yaml.org,2002:str) \n- !!null ~\n- !!str ~", + output.data \ No newline at end of file