* only run `teardown` if `setup` completed
With #35, it is no longer possible in `teardown` to refer to variables
introduced in `setup`. To avoid having to rewrite tests, address the
issue in #35 by wrapping `body` with an additional exception catching
layer. This then allows to re-use the previous `teardown` semantics
where `teardown` only ran if `setup` fully completed and also has access
to the variables introduced in `setup`.
* workaround shadowing bug
* Add test
---------
Co-authored-by: jangko <jangko128@gmail.com>
* unittest2.nim: ensure the testTeardownIMPL is performed at the end
This is important to prevent failing tests
to fail in an uncontrolled way and generating a SIGSEGV
(segmentation fault)
Particularly, when the test body raises an exception,
e.g. "assert false", what happened was that the "testTeardownIMPL"
was invoked before the exception handlging and that caused
errors like:
Traceback (most recent call last, using override)
/home/ivansete/workspace/status/nwaku/vendor/nim-unittest2/unittest2.nim(1154)
unittest2
/home/ivansete/workspace/status/nwaku/vendor/nim-unittest2/unittest2.nim(1086)
runDirect
/home/ivansete/workspace/status/nwaku/vendor/nim-unittest2/unittest2.nim(1111)
runTestX60gensym398
/home/ivansete/workspace/status/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/excpt.nim(631)
signalHandler
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Segmentation fault (core dumped)
This change brings 3 new items to `unittest2`:
* `-d:unittest2Static` compile-time flag that enables `test` to run both
at compile time and runtime
* `staticTest` that only run at compilet ime no matter the flag
* `runtimeTest` that only run at run time no matter the flag
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
* add counters to xml reports
* add command line args to write xml report
* fix initialization being performed once for each test
* use monotonic timer for duration measurements
Etan Kissling
tersec
jangko
Ivan FB
Jacek Sieka
Tanguy
narimiran
diegomrsantos
Ivan Yonchovski
Ștefan Talpalaru