status-im/NimYAML
2
0
Fork 0
mirror of https://github.com/status-im/NimYAML.git synced 2025-01-26 19:19:24 +00:00
Code Issues Projects Releases Wiki Activity

Better doc index. YamlStream now an object.

Browse Source 
* Also various fixes to serialization and presentation
This commit is contained in:
Felix Krause 2016-02-12 19:53:25 +01:00
parent cb18c5cb9c
commit c6c13eb044
11 changed files with 452 additions and 299 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

146
doc/index.txt
View File

@ -2,18 +2,146 @@
NimYAML
=======
Overview
========
Introduction
============
**NimYAML** is a pure YAML implementation for Nim. It is able to read from and
write to YAML character streams, and to serialize from and construct to native
Nim types. It exclusively supports
`YAML 1.2 <#http://www.yaml.org/spec/1.2/spec.html>`_. NimYAML exposes all
processing steps defined by the YAML specification to the user, so it is
possible to customize the processing pipeline. The following diagram gives an
overview of NimYAML's features based on the YAML processing pipeline. The items
and terminology YAML defines is shown in black, NimYAML's implementation is
shown in red.
`YAML 1.2 <#http://www.yaml.org/spec/1.2/spec.html>`_.
Quickstart
----------
Data used in quickstart code may not be accurate and is solely used to showcase
NimYAML's features.
.. code-block:: nim
import yaml
type
GenderKind* = enum
male, female, other
Person* = object
name*: string
gender*: GenderKind
age*: int32
spouse*: ref Person
offspring*: seq[ref Person]
let input = newStringStream("""
%YAML 1.2
---
- &a
name: Luke Skywalker
gender: male
age: 19
spouse: ~
offspring: []
- &b
name: Han Solo
gender: male
age: 35
spouse: &c
name: Leia Organa
gender: female
age: 19
spouse: *b
offspring: []
offspring: []
- *c
-
name: Anakin Skywalker
gender: male
age: 42
spouse: ~
offspring: [*a, *c]
""")
var persons: seq[ref Person]
load(input, persons)
for person in persons:
echo person.name, "\nage ", person.age
if person.spouse != nil:
echo "spouse: ", person.spouse.name
for child in person.offspring:
case child.gender
of male: echo "son: ", child.name
of female: echo "daughter: ", child.name
of other: echo "child: ", child.name
echo "------------------------"
dump(persons, newFileStream(stdout))
API Overview
============
NimYAML advocates parsing YAML input into native Nim types. Basic library types
like integers, floats and strings, as well as all tuples, enums and objects
without private fields are supported out-of-the-box. Reference types are also
supported, and NimYAML is able to detect if a reference occurs more than once
and will serialize it accordingly. This means that NimYAML is able to dump and
load potentially cyclic objects.
While loading into and dumping from native Nim types is the preferred way to use
NimYAML, it also gives you complete control over each processing step, so that
you can for example only use the parser and process its event stream yourself.
The following diagram gives an overview of NimYAML's features based on the YAML
processing pipeline. The items and terminology YAML defines is shown in
*italic*, NimYAML's implementation name is shown in **bold**.
.. image:: processing.svg
:align: center
Intermediate Representation
---------------------------
The base of all YAML processing with NimYAML is the
`YamlStream <yaml.html#YamlStream>`_. This is basically an iterator over
`YamlStreamEvent <yaml.html#YamlStreamEvent>`_ objects. Every proc that
represents a single stage of the loading or dumping process will either take a
``YamlStream`` as input or return a ``YamlStream``. Procs that implement the
whole process in one step hide the ``YamlStream`` from the user. Every proc that
returns a ``YamlStream`` guarantees that this stream is well-formed according to
the YAML specification.
This stream-oriented API can efficiently be used to parse large amounts of data.
The drawback is that errors in the input are only discovered while processing
the ``YamlStream``. If the ``YamlStream`` encounters an exception while
producing the next event, it will throw a ``YamlStreamError`` which contains the
original exception as ``parent``. The caller should know which exceptions are
possible as parents of ``YamlStream`` because they know the source of the
``YamlStream`` they provided.
Loading YAML
------------
For parsing, a `YamlParser <yaml.html#YamlParser>`_ object is needed. This
object stores some state while parsing that may be useful for error reporting to
the user. The
`parse <yaml.html#present,YamlStream,Stream,TagLibrary,PresentationStyle,int>`_
proc implements the YAML processing step of the same name. All syntax errors in
the input character stream are processed by ``parse``, which will raise a
``YamlParserError`` if it encounters a syntax error.
Transforming a ``YamlStream`` to a native YAML object is done via
``construct``. It skips the ``compose`` step for efficiency reasons. As Nim is
statically typed, you have to know the target type when you write your loading
code. This is different from YAML APIs of dynamically typed languages. If you
cannot know the type of your YAML input at compile time, you have to manually
process the ``YamlStream`` to serve your needs.
If you want to load YAML character data directly into a native Nim variable, you
can use `load <yaml.html#load,Stream,K>`_.
Dumping YAML
------------
Dumping YAML is straightforward: You transform a variable into a ``YamlStream``
with `represent <yaml.html#represent,T,TagStyle,AnchorStyle>`_ and then write
that to a stream using
`present <yaml.html#present,YamlStream,Stream,TagLibrary,PresentationStyle,int>`_.
If you want to execute both steps at once, you can use
`dump <yaml.html#dump,K,Stream,PresentationStyle,TagStyle,AnchorStyle,int>`_.
The ``present`` step allows you to specify how you want the output YAML
character stream to be formatted. Amongst other options, it is possible to
output pure JSON, but only if the stream does not contain any constructs that
cannot be presented in JSON.

6
doc/style.css
View File

@ -90,4 +90,10 @@ dt:before {
#testingground .style-option {
display: inline-block;
margin-right: 20px;
}
object {
margin-left: auto;
margin-right: auto;
display: block;
}

5
private/fastparse.nim
View File

@ -986,7 +986,7 @@ template blockScalar(lexer: BaseLexer, content: var string,
discard
proc parse*(p: YamlParser, s: Stream): YamlStream =
result = iterator(): YamlStreamEvent =
var backend = iterator(): YamlStreamEvent =
var
state = fpInitial
shorthands: Table[string, string]
@ -1600,3 +1600,6 @@ proc parse*(p: YamlParser, s: Stream): YamlStream =
else:
startToken()
parserError("Unexpected content (expected flow indicator)")
try:
result = initYamlStream(backend)
except Exception: assert(false) # compiler error

19
private/json.nim
View File

@ -78,26 +78,13 @@ proc jsonFromScalar(content: string, tag: TagId): JsonNode
e.parent = getCurrentException()
raise e
proc constructJson*(s: YamlStream): seq[JsonNode] =
proc constructJson*(s: var YamlStream): seq[JsonNode] =
newSeq(result, 0)
var
levels = newSeq[Level]()
anchors = initTable[AnchorId, JsonNode]()
safeIter = iterator(): YamlStreamEvent
{.raises: [YamlConstructionStreamError].} =
while true:
var item: YamlStreamEvent
try:
item = s()
if finished(s): break
except AssertionError: raise
except Exception:
var e = newException(YamlConstructionStreamError, "")
e.parent = getCurrentException()
raise e
yield item
for event in safeIter():
for event in s:
case event.kind
of yamlStartDocument:
# we don't need to do anything here; root node will be created
@ -205,7 +192,7 @@ proc loadToJson*(s: Stream): seq[JsonNode] =
e.column = parser.getColNumber()
e.lineContent = parser.getLineContent()
raise e
except YamlConstructionStreamError:
except YamlStreamError:
let e = getCurrentException()
if e.parent of IOError:
raise cast[ref IOError](e.parent)

103
private/presenter.nim
View File

@ -160,28 +160,16 @@ proc writeTagAndAnchor(target: Stream, tag: TagId, tagLib: TagLibrary,
e.parent = getCurrentException()
raise e
proc present*(s: YamlStream, target: Stream, tagLib: TagLibrary,
proc present*(s: var YamlStream, target: Stream, tagLib: TagLibrary,
style: PresentationStyle = psDefault,
indentationStep: int = 2) =
var
cached = initQueue[YamlStreamEvent]()
cacheIterator = iterator(): YamlStreamEvent =
while true:
while cached.len > 0:
yield cached.dequeue()
try:
let item = s()
if finished(s):
break
cached.enqueue(item)
except Exception:
var e = newException(YamlPresenterStreamError, "")
e.parent = getCurrentException()
raise e
indentation = 0
levels = newSeq[DumperState]()
cached = initQueue[YamlStreamEvent]()
for item in cacheIterator():
while cached.len > 0 or not s.finished():
let item = if cached.len > 0: cached.dequeue else: s.next()
case item.kind
of yamlStartDocument:
if style != psJson:
@ -253,24 +241,19 @@ proc present*(s: YamlStream, target: Stream, tagLib: TagLibrary,
of psDefault:
var length = 0
while true:
try:
let next = s()
assert (not finished(s))
cached.enqueue(next)
case next.kind
of yamlScalar:
length += 2 + next.scalarContent.len
of yamlAlias:
length += 6
of yamlEndSequence:
break
else:
length = high(int)
break
except Exception:
var e = newException(YamlPresenterStreamError, "")
e.parent = getCurrentException()
raise e
assert(not(s.finished()))
let next = s.next()
cached.enqueue(next)
case next.kind
of yamlScalar:
length += 2 + next.scalarContent.len
of yamlAlias:
length += 6
of yamlEndSequence:
break
else:
length = high(int)
break
nextState = if length <= 60: dFlowSequenceStart else:
dBlockSequenceItem
of psJson:
@ -318,24 +301,16 @@ proc present*(s: YamlStream, target: Stream, tagLib: TagLibrary,
mpInitial, mpKey, mpValue, mpNeedBlock
var mps = mpInitial
while mps != mpNeedBlock:
try:
let next = s()
assert (not finished(s))
cached.enqueue(next)
case next.kind
of yamlScalar, yamlAlias:
case mps
of mpInitial: mps = mpKey
of mpKey: mps = mpValue
else: mps = mpNeedBlock
of yamlEndMap:
break
else:
mps = mpNeedBlock
except Exception:
var e = newException(YamlPresenterStreamError, "")
e.parent = getCurrentException()
raise e
case s.peek().kind
of yamlScalar, yamlAlias:
case mps
of mpInitial: mps = mpKey
of mpKey: mps = mpValue
else: mps = mpNeedBlock
of yamlEndMap:
break
else:
mps = mpNeedBlock
nextState = if mps == mpNeedBlock: dBlockMapValue else:
dBlockInlineMap
of psMinimal:
@ -363,11 +338,11 @@ proc present*(s: YamlStream, target: Stream, tagLib: TagLibrary,
indentation += indentationStep
else:
if nextState in [dBlockMapValue, dBlockImplicitMapKey]:
startItem(target, style, indentation, levels[levels.high],
true)
if style != psJson:
writeTagAndAnchor(target,
item.mapTag, tagLib, item.mapAnchor)
startItem(target, style, indentation, levels[levels.high],
true)
else:
startItem(target, style, indentation, levels[levels.high],
true)
@ -453,15 +428,7 @@ proc present*(s: YamlStream, target: Stream, tagLib: TagLibrary,
assert false
indentation -= indentationStep
of yamlEndDocument:
try:
let next = s()
if finished(s):
break
cached.enqueue(next)
except Exception:
var e = newException(YamlPresenterStreamError, "")
e.parent = getCurrentException()
raise e
if finished(s): break
safeWrite("...\x0A")
proc transform*(input: Stream, output: Stream, style: PresentationStyle,
@ -473,7 +440,7 @@ proc transform*(input: Stream, output: Stream, style: PresentationStyle,
try:
if style == psCanonical:
var specificTagEvents = iterator(): YamlStreamEvent =
for e in events():
for e in events:
var event = e
case event.kind
of yamlStartDocument, yamlEndDocument, yamlEndMap,
@ -503,12 +470,14 @@ proc transform*(input: Stream, output: Stream, style: PresentationStyle,
elif event.scalarTag == yTagExclamationMark:
event.scalarTag = yTagString
yield event
present(specificTagEvents, output, tagLib, style,
var s = initYamlStream(specificTagEvents)
present(s, output, tagLib, style,
indentationStep)
else:
present(events, output, tagLib, style, indentationStep)
except YamlPresenterStreamError:
let e = getCurrentException()
except YamlStreamError:
var e = getCurrentException()
while e.parent of YamlStreamError: e = e.parent
if e.parent of IOError:
raise cast[ref IOError](e.parent)
elif e.parent of YamlParserError:

55
private/streams.nim Normal file
View File

@ -0,0 +1,55 @@
proc initYamlStream*(backend: iterator(): YamlStreamEvent): YamlStream =
result.peeked = false
result.backend = backend
proc next*(s: var YamlStream): YamlStreamEvent =
if s.peeked:
s.peeked = false
result = s.cached
else:
try:
result = s.backend()
assert(not finished(s.backend))
except AssertionError: raise
except YamlStreamError:
let cur = getCurrentException()
var e = newException(YamlStreamError, cur.msg)
e.parent = cur.parent
raise e
except Exception:
let cur = getCurrentException()
var e = newException(YamlStreamError, cur.msg)
e.parent = cur
raise e
proc peek*(s: var YamlStream): YamlStreamEvent =
if not s.peeked:
s.cached = s.next()
s.peeked = true
result = s.cached
proc `peek=`*(s: var YamlStream, value: YamlStreamEvent) =
s.cached = value
s.peeked = true
proc finished*(s: var YamlStream): bool =
if s.peeked:
result = false
else:
try:
s.cached = s.backend()
if finished(s.backend): result = true
else:
s.peeked = true
result = false
except AssertionError: raise
except YamlStreamError:
let cur = getCurrentException()
var e = newException(YamlStreamError, cur.msg)
e.parent = cur.parent
raise e
except Exception:
let cur = getCurrentException()
var e = newException(YamlStreamError, cur.msg)
e.parent = cur
raise e

3
test/constructingJson.nim
View File

@ -8,7 +8,8 @@ proc wc(line, column: int, lineContent: string, message: string) =
proc ensureEqual(yamlIn, jsonIn: string) =
var
parser = newYamlParser(initCoreTagLibrary(), wc)
yamlResult = constructJson(parser.parse(newStringStream(yamlIn)))
s = parser.parse(newStringStream(yamlIn))
yamlResult = constructJson(s)
jsonResult = parseJson(jsonIn)
assert yamlResult.len == 1
assert(jsonResult == yamlResult[0])

2
test/parsing.nim
View File

@ -56,7 +56,7 @@ template ensure(input: string, expected: varargs[YamlStreamEvent]) {.dirty.} =
parser = newYamlParser(tagLib)
events = parser.parse(newStringStream(input))
try:
for token in events():
for token in events:
if i >= expected.len:
echo "received more tokens than expected (next token = ",
token.kind, ")"

9
test/serializing.nim
View File

@ -223,7 +223,14 @@ next:
result: seq[ref Node]
parser = newYamlParser(tagLib)
events = parser.parse(input)
construct(events, result)
try:
construct(events, result)
except YamlConstructionError:
let ex = (ref YamlConstructionError)(getCurrentException())
echo "line ", parser.getLineNumber, ", column ",
parser.getColNumber, ": ", ex.msg
echo parser.getLineContent
assert(result.len == 3)
assert(result[0].value == "a")
assert(result[1].value == "b")

72
yaml.nim
View File

@ -100,8 +100,9 @@ type
of yamlAlias:
aliasTarget* : AnchorId
YamlStream* = iterator(): YamlStreamEvent ## \
## A ``YamlStream`` is an iterator that yields a well-formed stream of
YamlStream* = object ## \
## A ``YamlStream`` is an iterator-like object that yields a
## well-formed stream of
## ``YamlStreamEvents``. Well-formed means that every ``yamlStartMap``
## is terminated by a ``yamlEndMap``, every ``yamlStartSequence`` is
## terminated by a ``yamlEndSequence`` and every ``yamlStartDocument``
@ -113,7 +114,12 @@ type
## and is not required to check for it. The procs in this module will
## always yield a well-formed ``YamlStream`` and expect it to be
## well-formed if it's an input.
##
##
backend: iterator(): YamlStreamEvent
peeked: bool
cached: YamlStreamEvent
TagLibrary* = ref object
## A ``TagLibrary`` maps tag URIs to ``TagId`` s.
##
@ -228,10 +234,9 @@ type
## writing character data to the output stream raises any exception.
## The error that has occurred is available from ``parent``.
YamlPresenterStreamError* = object of Exception
## Exception that may be raised by the YAML presenter. This occurs if
## an exception is raised while retrieving the next item from a
## `YamlStream <#YamlStream>`_. The error that has occurred is
YamlStreamError* = object of Exception
## Exception that may be raised by a ``YamlStream`` when the underlying
## backend raises an exception. The error that has occurred is
## available from ``parent``.
YamlConstructionError* = object of YamlLoadingError
@ -240,11 +245,7 @@ type
## ``lineContent`` are only available if the costructing proc also does
## parsing, because otherwise this information is not available to the
## costruction proc.
YamlConstructionStreamError* = object of YamlLoadingError
## Exception that may be raised by a constructor if the input
## `YamlStream <#YamlStream>`_ raises an error. The error that has
## occurred is available from ``parent``.
const
# failsafe schema
@ -327,6 +328,33 @@ proc `==`*(left, right: AnchorId): bool {.borrow.}
proc `$`*(id: AnchorId): string {.borrow.}
proc hash*(id: AnchorId): Hash {.borrow.}
proc initYamlStream*(backend: iterator(): YamlStreamEvent):
YamlStream {.raises: [].}
proc next*(s: var YamlStream): YamlStreamEvent {.raises: [YamlStreamError].}
proc peek*(s: var YamlStream): YamlStreamEvent {.raises: [YamlStreamError].}
proc `peek=`*(s: var YamlStream, value: YamlStreamEvent) {.raises: [].}
proc finished*(s: var YamlStream): bool {.raises: [YamlStreamError].}
iterator items*(s: var YamlStream): YamlStreamEvent {.raises: [YamlStreamError].} =
if s.peeked:
s.peeked = false
yield s.cached
while true:
var event: YamlStreamEvent
try:
event = s.backend()
if finished(s.backend): break
except AssertionError: raise
except YamlStreamError:
let cur = getCurrentException()
var e = newException(YamlStreamError, cur.msg)
e.parent = cur.parent
raise e
except Exception:
var e = newException(YamlStreamError, getCurrentExceptionMsg())
e.parent = getCurrentException()
raise e
yield event
proc initTagLibrary*(): TagLibrary {.raises: [].}
## initializes the ``tags`` table and sets ``nextCustomTagId`` to
## ``yFirstCustomTagId``.
@ -386,12 +414,11 @@ proc getLineContent*(p: YamlParser, marker: bool = true): string {.raises: [].}
## be returned containing a ``^`` at the position of the recent parser
## token.
proc parse*(p: YamlParser, s: Stream):
YamlStream {.raises: [IOError, YamlParserError].}
proc parse*(p: YamlParser, s: Stream): YamlStream {.raises: [].}
## Parse the given stream as YAML character stream.
proc constructJson*(s: YamlStream): seq[JsonNode]
{.raises: [YamlConstructionError, YamlConstructionStreamError].}
proc constructJson*(s: var YamlStream): seq[JsonNode]
{.raises: [YamlConstructionError, YamlStreamError].}
## Construct an in-memory JSON tree from a YAML event stream. The stream may
## not contain any tags apart from those in ``coreTagLibrary``. Anchors and
## aliases will be resolved. Maps in the input must not contain
@ -406,22 +433,24 @@ proc constructJson*(s: YamlStream): seq[JsonNode]
## of these values into a JSON character stream.
proc loadToJson*(s: Stream): seq[JsonNode]
{.raises: [IOError, YamlParserError, YamlConstructionError].}
{.raises: [IOError, YamlParserError, YamlConstructionError,
OutOfMemError].}
## Uses `YamlParser <#YamlParser>`_ and
## `constructJson <#constructJson>`_ to construct an in-memory JSON tree
## from a YAML character stream.
proc present*(s: YamlStream, target: Stream, tagLib: TagLibrary,
proc present*(s: var YamlStream, target: Stream, tagLib: TagLibrary,
style: PresentationStyle = psDefault,
indentationStep: int = 2) {.raises: [YamlPresenterJsonError,
YamlPresenterOutputError,
YamlPresenterStreamError].}
YamlStreamError].}
## Convert ``s`` to a YAML character stream and write it to ``target``.
proc transform*(input: Stream, output: Stream, style: PresentationStyle,
indentationStep: int = 2) {.raises: [IOError, YamlParserError,
YamlPresenterJsonError,
YamlPresenterOutputError].}
YamlPresenterOutputError,
OutOfMemError].}
## Parser ``input`` as YAML character stream and then dump it to ``output``
## while resolving non-specific tags to the ones in the YAML core tag
## library.
@ -433,4 +462,5 @@ include private.events
include private.json
include private.presenter
include private.hints
include private.fastparse
include private.fastparse
include private.streams

331
yaml/serialization.nim
View File

@ -21,6 +21,8 @@ type
refsList: seq[RefNodeData]
style: AnchorStyle
RawYamlStream* = iterator(): YamlStreamEvent {.raises: [].}
const
yTagNimInt8* = 100.TagId
yTagNimInt16* = 101.TagId
@ -175,7 +177,7 @@ macro serializable*(types: stmt): stmt =
newNimNode(nnkExprColonExpr).add(newIdentNode("raises"),
newNimNode(nnkBracket).add(
newIdentNode("YamlConstructionError"),
newIdentNode("YamlConstructionStreamError"))))
newIdentNode("YamlStreamError"))))
impl = quote do:
var event = s()
if finished(s) or event.kind != yamlStartMap:
@ -213,7 +215,7 @@ macro serializable*(types: stmt): stmt =
# representObject()
var representProc = newProc(newIdentNode("representObject"), [
newIdentNode("YamlStream"),
newIdentNode("RawYamlStream"),
newIdentDefs(newIdentNode("value"), tIdent),
newIdentDefs(newIdentNode("ts"),
newIdentNode("TagStyle")),
@ -278,12 +280,6 @@ macro serializable*(types: stmt): stmt =
representProc[6] = impl
result.add(representProc)
proc prepend(event: YamlStreamEvent, s: YamlStream): YamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield event
for e in s():
yield e
proc safeTagUri*(id: TagId): string {.raises: [].} =
try:
let uri = serializationTagLibrary.uri(id)
@ -295,17 +291,10 @@ proc safeTagUri*(id: TagId): string {.raises: [].} =
# cannot happen (theoretically, you known)
assert(false)
template constructScalarItem(item: YamlStreamEvent, name: string, t: TagId,
content: stmt) =
try:
item = s()
except YamlConstructionStreamError, AssertionError:
raise
except Exception:
var e = newException(YamlConstructionStreamError, "")
e.parent = getCurrentException()
raise e
if finished(s) or item.kind != yamlScalar:
template constructScalarItem(bs: var YamlStream, item: YamlStreamEvent,
name: string, t: TagId, content: stmt) =
item = bs.next()
if item.kind != yamlScalar:
raise newException(YamlConstructionError, "Expected scalar")
if item.scalarTag notin [yTagQuestionMark, yTagExclamationMark, t]:
raise newException(YamlConstructionError, "Wrong tag for " & name)
@ -319,26 +308,17 @@ template constructScalarItem(item: YamlStreamEvent, name: string, t: TagId,
e.parent = getCurrentException()
raise e
template safeNextEvent(e: YamlStreamEvent, s: YamlStream) =
try:
e = s()
except YamlConstructionStreamError, AssertionError:
raise
except Exception:
var ex = newException(YamlConstructionStreamError, "")
ex.parent = getCurrentException()
raise ex
proc yamlTag*(T: typedesc[string]): TagId {.inline, raises: [].} = yTagString
proc constructObject*(s: YamlStream, c: ConstructionContext, result: var string)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
proc constructObject*(s: var YamlStream, c: ConstructionContext,
result: var string)
{.raises: [YamlConstructionError, YamlStreamError].} =
var item: YamlStreamEvent
constructScalarItem(item, "string", yTagString):
constructScalarItem(s, item, "string", yTagString):
result = item.scalarContent
proc representObject*(value: string, ts: TagStyle = tsNone,
c: SerializationContext): YamlStream {.raises: [].} =
c: SerializationContext): RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield scalarEvent(value, presentTag(string, ts), yAnchorNone)
@ -348,25 +328,25 @@ proc yamlTag*(T: typedesc[int32]): TagId {.inline, raises: [].} = yTagNimInt32
proc yamlTag*(T: typedesc[int64]): TagId {.inline, raises: [].} = yTagNimInt64
proc constructObject*[T: int8|int16|int32|int64](
s: YamlStream, c: ConstructionContext, result: var T)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
s: var YamlStream, c: ConstructionContext, result: var T)
{.raises: [YamlConstructionError, YamlStreamError].} =
var item: YamlStreamEvent
constructScalarItem(item, name(T), yamlTag(T)):
constructScalarItem(s, item, name(T), yamlTag(T)):
result = T(parseBiggestInt(item.scalarContent))
template constructObject*(s: YamlStream, c: ConstructionContext,
template constructObject*(s: var YamlStream, c: ConstructionContext,
result: var int) =
{.fatal: "The length of `int` is platform dependent. Use int[8|16|32|64].".}
discard
proc representObject*[T: int8|int16|int32|int64](
value: T, ts: TagStyle = tsNone, c: SerializationContext):
YamlStream {.raises: [].} =
RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield scalarEvent($value, presentTag(T, ts), yAnchorNone)
template representObject*(value: int, tagStyle: TagStyle,
c: SerializationContext) =
c: SerializationContext): RawYamlStream =
{.fatal: "The length of `int` is platform dependent. Use int[8|16|32|64].".}
discard
@ -388,13 +368,13 @@ proc parseBiggestUInt(s: string): uint64 =
{.pop.}
proc constructObject*[T: uint8|uint16|uint32|uint64](
s: YamlStream, c: ConstructionContext, result: var T)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
s: var YamlStream, c: ConstructionContext, result: var T)
{.raises: [YamlConstructionError, YamlStreamError].} =
var item: YamlStreamEvent
constructScalarItem(item, name[T], yamlTag(T)):
constructScalarItem(s, item, name[T], yamlTag(T)):
result = T(parseBiggestUInt(item.scalarContent))
template constructObject*(s: YamlStream, c: ConstructionContext,
template constructObject*(s: var YamlStream, c: ConstructionContext,
result: var uint) =
{.fatal:
"The length of `uint` is platform dependent. Use uint[8|16|32|64].".}
@ -402,11 +382,12 @@ template constructObject*(s: YamlStream, c: ConstructionContext,
proc representObject*[T: uint8|uint16|uint32|uint64](
value: T, ts: TagStyle, c: SerializationContext):
YamlStream {.raises: [].} =
RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield scalarEvent($value, presentTag(T, ts), yAnchorNone)
template representObject*(value: uint, ts: TagStyle, c: SerializationContext) =
template representObject*(value: uint, ts: TagStyle, c: SerializationContext):
RawYamlStream =
{.fatal:
"The length of `uint` is platform dependent. Use uint[8|16|32|64].".}
discard
@ -417,10 +398,10 @@ proc yamlTag*(T: typedesc[float64]): TagId {.inline, raises: [].} =
yTagNimFloat64
proc constructObject*[T: float32|float64](
s: YamlStream, c: ConstructionContext, result: var T)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
s: var YamlStream, c: ConstructionContext, result: var T)
{.raises: [YamlConstructionError, YamlStreamError].} =
var item: YamlStreamEvent
constructScalarItem(item, name(T), yamlTag(T)):
constructScalarItem(s, item, name(T), yamlTag(T)):
let hint = guessType(item.scalarContent)
case hint
of yTypeFloat:
@ -436,13 +417,13 @@ proc constructObject*[T: float32|float64](
raise newException(YamlConstructionError,
"Cannot construct to float: " & item.scalarContent)
template constructObject*(s: YamlStream, c: ConstructionContext,
template constructObject*(s: var YamlStream, c: ConstructionContext,
result: var float) =
{.fatal: "The length of `float` is platform dependent. Use float[32|64].".}
proc representObject*[T: float32|float64](value: T, ts: TagStyle,
c: SerializationContext):
YamlStream {.raises: [].} =
RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
var
asString: string
@ -458,15 +439,16 @@ proc representObject*[T: float32|float64](value: T, ts: TagStyle,
yield scalarEvent(asString, presentTag(T, ts), yAnchorNone)
template representObject*(value: float, tagStyle: TagStyle,
c: SerializationContext) =
c: SerializationContext): RawYamlStream =
{.fatal: "The length of `float` is platform dependent. Use float[32|64].".}
proc yamlTag*(T: typedesc[bool]): TagId {.inline, raises: [].} = yTagBoolean
proc constructObject*(s: YamlStream, c: ConstructionContext, result: var bool)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
proc constructObject*(s: var YamlStream, c: ConstructionContext,
result: var bool)
{.raises: [YamlConstructionError, YamlStreamError].} =
var item: YamlStreamEvent
constructScalarItem(item, "bool", yTagBoolean):
constructScalarItem(s, item, "bool", yTagBoolean):
case guessType(item.scalarContent)
of yTypeBoolTrue:
result = true
@ -477,17 +459,18 @@ proc constructObject*(s: YamlStream, c: ConstructionContext, result: var bool)
"Cannot construct to bool: " & item.scalarContent)
proc representObject*(value: bool, ts: TagStyle,
c: SerializationContext): YamlStream {.raises: [].} =
c: SerializationContext): RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield scalarEvent(if value: "y" else: "n", presentTag(bool, ts),
yAnchorNone)
proc yamlTag*(T: typedesc[char]): TagId {.inline, raises: [].} = yTagNimChar
proc constructObject*(s: YamlStream, c: ConstructionContext, result: var char)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
proc constructObject*(s: var YamlStream, c: ConstructionContext,
result: var char)
{.raises: [YamlConstructionError, YamlStreamError].} =
var item: YamlStreamEvent
constructScalarItem(item, "char", yTagNimChar):
constructScalarItem(s, item, "char", yTagNimChar):
if item.scalarContent.len != 1:
raise newException(YamlConstructionError,
"Cannot construct to char (length != 1): " &
@ -496,7 +479,7 @@ proc constructObject*(s: YamlStream, c: ConstructionContext, result: var char)
result = item.scalarContent[0]
proc representObject*(value: char, ts: TagStyle,
c: SerializationContext): YamlStream {.raises: [].} =
c: SerializationContext): RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield scalarEvent("" & value, presentTag(char, ts), yAnchorNone)
@ -504,35 +487,28 @@ proc yamlTag*[I](T: typedesc[seq[I]]): TagId {.inline, raises: [].} =
let uri = "!nim:system:seq(" & safeTagUri(yamlTag(I)) & ")"
result = lazyLoadTag(uri)
proc constructObject*[T](s: YamlStream, c: ConstructionContext,
proc constructObject*[T](s: var YamlStream, c: ConstructionContext,
result: var seq[T])
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
var event: YamlStreamEvent
safeNextEvent(event, s)
if finished(s) or event.kind != yamlStartSequence:
{.raises: [YamlConstructionError, YamlStreamError].} =
let event = s.next()
if event.kind != yamlStartSequence:
raise newException(YamlConstructionError, "Expected sequence start")
if event.seqTag notin [yTagQuestionMark, yamlTag(seq[T])]:
raise newException(YamlConstructionError, "Wrong tag for seq[T]")
result = newSeq[T]()
safeNextEvent(event, s)
assert(not finished(s))
while event.kind != yamlEndSequence:
var
item: T
events = prepend(event, s)
try:
constructObject(events, c, item)
while s.peek().kind != yamlEndSequence:
var item: T
try: constructObject(s, c, item)
except AssertionError, YamlConstructionError,
YamlConstructionStreamError: raise
YamlStreamError: raise
except:
# compiler bug: https://github.com/nim-lang/Nim/issues/3772
assert(false)
result.add(item)
safeNextEvent(event, s)
assert(not finished(s))
discard s.next()
proc representObject*[T](value: seq[T], ts: TagStyle,
c: SerializationContext): YamlStream {.raises: [].} =
c: SerializationContext): RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
let childTagStyle = if ts == tsRootOnly: tsNone else: ts
yield YamlStreamEvent(kind: yamlStartSequence,
@ -540,7 +516,9 @@ proc representObject*[T](value: seq[T], ts: TagStyle,
seqAnchor: yAnchorNone)
for item in value:
var events = representObject(item, childTagStyle, c)
for event in events():
while true:
let event = events()
if finished(events): break
yield event
yield YamlStreamEvent(kind: yamlEndSequence)
@ -559,38 +537,32 @@ proc yamlTag*[K, V](T: typedesc[Table[K, V]]): TagId {.inline, raises: [].} =
# cannot happen (theoretically, you known)
assert(false)
proc constructObject*[K, V](s: YamlStream, c: ConstructionContext,
proc constructObject*[K, V](s: var YamlStream, c: ConstructionContext,
result: var Table[K, V])
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
var event: YamlStreamEvent
safeNextEvent(event, s)
assert(not finished(s))
{.raises: [YamlConstructionError, YamlStreamError].} =
let event = s.next()
if event.kind != yamlStartMap:
raise newException(YamlConstructionError, "Expected map start, got " &
$event.kind)
if event.mapTag notin [yTagQuestionMark, yamlTag(Table[K, V])]:
raise newException(YamlConstructionError, "Wrong tag for Table[K, V]")
result = initTable[K, V]()
safeNextEvent(event, s)
assert(not finished(s))
while event.kind != yamlEndMap:
while s.peek.kind != yamlEndMap:
var
key: K
value: V
events = prepend(event, s)
try:
constructObject(events, c, key)
constructObject(s, c, key)
constructObject(s, c, value)
except AssertionError: raise
except Exception:
# compiler bug: https://github.com/nim-lang/Nim/issues/3772
assert(false)
result[key] = value
safeNextEvent(event, s)
assert(not finished(s))
discard s.next()
proc representObject*[K, V](value: Table[K, V], ts: TagStyle,
c: SerializationContext): YamlStream {.raises:[].} =
c: SerializationContext): RawYamlStream {.raises:[].} =
result = iterator(): YamlStreamEvent =
let childTagStyle = if ts == tsRootOnly: tsNone else: ts
yield YamlStreamEvent(kind: yamlStartMap,
@ -598,16 +570,20 @@ proc representObject*[K, V](value: Table[K, V], ts: TagStyle,
mapAnchor: yAnchorNone)
for key, value in value.pairs:
var events = representObject(key, childTagStyle, c)
for event in events():
while true:
let event = events()
if finished(events): break
yield event
events = representObject(value, childTagStyle, c)
for event in events():
while true:
let event = events()
if finished(events): break
yield event
yield YamlStreamEvent(kind: yamlEndMap)
template yamlTag*(T: typedesc[object|enum]): expr =
var uri = when compiles(yamlTagId(T)): yamlTagId(T) else:
"!nim:custom:" & T.name
"!nim:custom:" & (typetraits.name(type(T)))
try:
serializationTagLibrary.tags[uri]
except KeyError:
@ -626,32 +602,34 @@ template yamlTag*(T: typedesc[tuple]): expr =
try: serializationTagLibrary.tags[uri]
except KeyError: serializationTagLibrary.registerUri(uri)
proc constructObject*[O: object|tuple](s: YamlStream, c: ConstructionContext,
result: var O)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
var e = s()
assert(not finished(s))
proc constructObject*[O](s: var YamlStream, c: ConstructionContext,
result: var ref O)
{.raises: [YamlConstructionError, YamlStreamError].}
proc constructObject*[O: object|tuple](s: var YamlStream,
c: ConstructionContext,
result: var O)
{.raises: [YamlConstructionError, YamlStreamError].} =
let e = s.next()
if e.kind != yamlStartMap:
raise newException(YamlConstructionError, "Expected map start, got " &
$e.kind)
if e.mapAnchor != yAnchorNone:
raise newException(YamlConstructionError, "Anchor on a non-ref type")
e = s()
assert(not finished(s))
while e.kind != yamlEndMap:
while s.peek.kind != yamlEndMap:
let e = s.next()
if e.kind != yamlScalar:
raise newException(YamlConstructionError, "Expected field name")
raise newException(YamlConstructionError,
"Expected field name, got " & $e.kind)
let name = e.scalarContent
for fname, value in fieldPairs(result):
if fname == name:
constructObject(s, c, value)
break
e = s()
assert(not finished(s))
discard s.next()
proc representObject*[O: object|tuple](value: O, ts: TagStyle,
c: SerializationContext):
YamlStream {.raises: [].} =
c: SerializationContext): RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
let childTagStyle = if ts == tsRootOnly: tsNone else: ts
yield startMapEvent(presentTag(O, ts), yAnchorNone)
@ -659,15 +637,16 @@ proc representObject*[O: object|tuple](value: O, ts: TagStyle,
yield scalarEvent(name, presentTag(string, childTagStyle),
yAnchorNone)
var events = representObject(value, childTagStyle, c)
for event in events():
while true:
let event = events()
if finished(events): break
yield event
yield endMapEvent()
proc constructObject*[O: enum](s: YamlStream, c: ConstructionContext,
proc constructObject*[O: enum](s: var YamlStream, c: ConstructionContext,
result: var O)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
let e = s()
assert(not finished(s))
{.raises: [YamlConstructionError, YamlStreamError].} =
let e = s.next()
if e.kind != yamlScalar:
raise newException(YamlConstructionError, "Expected scalar, got " &
$e.kind)
@ -679,55 +658,57 @@ proc constructObject*[O: enum](s: YamlStream, c: ConstructionContext,
raise ex
proc representObject*[O: enum](value: O, ts: TagStyle,
c: SerializationContext):
YamlStream {.raises: [].} =
c: SerializationContext): RawYamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield scalarEvent($value, presentTag(O, ts), yAnchorNone)
proc yamlTag*[O](T: typedesc[ref O]): TagId {.inline, raises: [].} = yamlTag(O)
proc constructObject*[O](s: YamlStream, c: ConstructionContext,
result: var ref O)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
var e = s()
assert(not finished(s))
proc constructObject*[O](s: var YamlStream, c: ConstructionContext,
result: var ref O) =
var e = s.peek()
if e.kind == yamlScalar:
if e.scalarTag == yTagNull or (
e.scalarTag in [yTagQuestionMark, yTagExclamationMark] and
guessType(e.scalarContent) == yTypeNull):
result = nil
discard s.next()
return
elif e.kind == yamlAlias:
try:
result = cast[ref O](c.refs[e.aliasTarget])
discard s.next()
return
except KeyError:
assert(false)
new(result)
var a: ptr AnchorId
template removeAnchor(anchor: var AnchorId) {.dirty.} =
if anchor != yAnchorNone:
assert(not c.refs.hasKey(anchor))
c.refs[anchor] = cast[pointer](result)
anchor = yAnchorNone
case e.kind
of yamlScalar: a = addr(e.scalarAnchor)
of yamlStartMap: a = addr(e.mapAnchor)
of yamlStartSequence: a = addr(e.seqAnchor)
of yamlScalar: removeAnchor(e.scalarAnchor)
of yamlStartMap: removeAnchor(e.mapAnchor)
of yamlStartSequence: removeAnchor(e.seqAnchor)
else: assert(false)
if a[] != yAnchorNone:
assert(not c.refs.hasKey(a[]))
c.refs[a[]] = cast[pointer](result)
a[] = yAnchorNone
s.peek = e
try:
constructObject(prepend(e, s), c, result[])
except YamlConstructionError, YamlConstructionStreamError, AssertionError:
constructObject(s, c, result[])
except YamlConstructionError, YamlStreamError, AssertionError:
raise
except Exception:
var e = newException(YamlConstructionStreamError,
var e = newException(YamlStreamError,
getCurrentExceptionMsg())
e.parent = getCurrentException()
raise e
proc representObject*[O](value: ref O, ts: TagStyle, c: SerializationContext):
YamlStream {.raises: [].} =
RawYamlStream {.raises: [].} =
if value == nil:
result = iterator(): YamlStreamEvent = yield scalarEvent("~", yTagNull)
result = iterator(): YamlStreamEvent =
yield scalarEvent("~", yTagNull)
elif c.style == asNone:
result = representObject(value[], ts, c)
else:
@ -743,39 +724,40 @@ proc representObject*[O](value: ref O, ts: TagStyle, c: SerializationContext):
c.refsList.add(initRefNodeData(p))
let a = if c.style == asAlways: AnchorId(c.refsList.high) else:
cast[AnchorId](p)
try:
var
objStream = representObject(value[], ts, c)
first = objStream()
assert(not finished(objStream))
case first.kind
result = iterator(): YamlStreamEvent =
var child = representObject(value[], ts, c)
var first = child()
assert(not finished(child))
case first.kind
of yamlStartMap:
first.mapAnchor = a
of yamlStartSequence:
first.seqAnchor = a
of yamlScalar:
first.scalarAnchor = a
else:
assert(false)
result = prepend(first, objStream)
except Exception:
assert(false)
else: discard
yield first
while true:
let event = child()
if finished(child): break
yield event
proc construct*[T](s: YamlStream, target: var T)
{.raises: [YamlConstructionError, YamlConstructionStreamError].} =
var context = newConstructionContext()
proc construct*[T](s: var YamlStream, target: var T)
{.raises: [YamlConstructionError, YamlStreamError].} =
var
context = newConstructionContext()
try:
var e = s()
assert((not finished(s)) and e.kind == yamlStartDocument)
var e = s.next()
assert(e.kind == yamlStartDocument)
constructObject(s, context, target)
e = s()
assert((not finished(s)) and e.kind == yamlEndDocument)
except YamlConstructionError, YamlConstructionStreamError, AssertionError:
e = s.next()
assert(e.kind == yamlEndDocument)
except YamlConstructionError, YamlStreamError, AssertionError:
raise
except Exception:
# may occur while calling s()
var ex = newException(YamlConstructionStreamError, "")
var ex = newException(YamlStreamError, "")
ex.parent = getCurrentException()
raise ex
@ -788,14 +770,13 @@ proc load*[K](input: Stream, target: var K)
construct(events, target)
except YamlConstructionError, AssertionError:
raise
except YamlConstructionStreamError:
let e = (ref YamlConstructionStreamError)(getCurrentException())
except YamlStreamError:
let e = (ref YamlStreamError)(getCurrentException())
if e.parent of IOError:
raise (ref IOError)(e.parent)
elif e.parent of YamlParserError:
raise (ref YamlParserError)(e.parent)
else:
echo e.parent.repr
assert(false)
except Exception:
# compiler bug: https://github.com/nim-lang/Nim/issues/3772
@ -825,32 +806,19 @@ proc setAliasAnchor(a: var AnchorId, q: var seq[RefNodeData]) {.inline.} =
a = q[i].anchor
return
assert(false)
proc insideDoc(s: YamlStream): YamlStream {.raises: [].} =
result = iterator(): YamlStreamEvent =
yield YamlStreamEvent(kind: yamlStartDocument)
while true:
var event: YamlStreamEvent
try:
event = s()
if finished(s): break
except AssertionError: raise
except Exception:
# serializing object does not raise any errors, so we can
# ignore this
assert(false)
yield event
yield YamlStreamEvent(kind: yamlEndDocument)
proc represent*[T](value: T, ts: TagStyle = tsRootOnly,
a: AnchorStyle = asTidy): YamlStream {.raises: [].} =
var
context = newSerializationContext(a)
objStream: YamlStream
try:
objStream = insideDoc(representObject(value, ts, context))
except Exception:
assert(false)
objStream = iterator(): YamlStreamEvent =
yield YamlStreamEvent(kind: yamlStartDocument)
var events = representObject(value, ts, context)
while true:
let e = events()
if finished(events): break
yield e
yield YamlStreamEvent(kind: yamlEndDocument)
if a == asTidy:
var objQueue = newSeq[YamlStreamEvent]()
try:
@ -859,7 +827,7 @@ proc represent*[T](value: T, ts: TagStyle = tsRootOnly,
except Exception:
assert(false)
var next = 0.AnchorId
result = iterator(): YamlStreamEvent =
var backend = iterator(): YamlStreamEvent =
for i in countup(0, objQueue.len - 1):
var event = objQueue[i]
case event.kind
@ -874,8 +842,9 @@ proc represent*[T](value: T, ts: TagStyle = tsRootOnly,
else:
discard
yield event
result = initYamlStream(backend)
else:
result = objStream
result = initYamlStream(objStream)
proc dump*[K](value: K, target: Stream, style: PresentationStyle = psDefault,
tagStyle: TagStyle = tsRootOnly,
@ -885,13 +854,11 @@ proc dump*[K](value: K, target: Stream, style: PresentationStyle = psDefault,
if style == psJson: asNone else: anchorStyle)
try:
present(events, target, serializationTagLibrary, style, indentationStep)
except YamlPresenterStreamError:
except YamlStreamError:
# serializing object does not raise any errors, so we can ignore this
var e = getCurrentException()
echo e.msg
echo e.parent.repr
assert(false)
except YamlPresenterJsonError, YamlPresenterOutputError, AssertionError:
except YamlPresenterJsonError, YamlPresenterOutputError, AssertionError, FieldError:
raise
except Exception:
# cannot occur as represent() doesn't raise any errors