2021-03-19 17:27:25 +00:00
|
|
|
#
|
|
|
|
#
|
|
|
|
# Nim's Runtime Library
|
|
|
|
# (c) Copyright 2015 Nim Contributors
|
|
|
|
# (c) Copyright 2019-2021 Ștefan Talpalaru
|
|
|
|
# (c) Copyright 2021-Onwards Status Research and Development
|
2019-05-19 21:11:17 +00:00
|
|
|
#
|
|
|
|
|
2023-07-28 03:53:13 +00:00
|
|
|
{.push raises: [].}
|
2021-03-19 17:27:25 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
## :Authors: Zahary Karadjov, Ștefan Talpalaru, Status Research and Development
|
2019-05-19 21:11:17 +00:00
|
|
|
##
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
## This module makes unit testing easy.
|
2019-05-19 21:11:17 +00:00
|
|
|
##
|
|
|
|
## .. code::
|
|
|
|
## nim c -r testfile.nim
|
|
|
|
##
|
|
|
|
## exits with 0 or 1.
|
|
|
|
##
|
|
|
|
## Running individual tests
|
|
|
|
## ========================
|
|
|
|
##
|
|
|
|
## Specify the test names as command line arguments.
|
|
|
|
##
|
|
|
|
## .. code::
|
|
|
|
##
|
|
|
|
## nim c -r test "my test name" "another test"
|
|
|
|
##
|
|
|
|
## Multiple arguments can be used.
|
|
|
|
##
|
|
|
|
## Running a single test suite
|
|
|
|
## ===========================
|
|
|
|
##
|
|
|
|
## Specify the suite name delimited by ``"::"``.
|
|
|
|
##
|
|
|
|
## .. code::
|
|
|
|
##
|
|
|
|
## nim c -r test "my suite name::"
|
|
|
|
##
|
|
|
|
## Selecting tests by pattern
|
|
|
|
## ==========================
|
|
|
|
##
|
|
|
|
## A single ``"*"`` can be used for globbing.
|
|
|
|
##
|
|
|
|
## Delimit the end of a suite name with ``"::"``.
|
|
|
|
##
|
|
|
|
## Tests matching **any** of the arguments are executed.
|
|
|
|
##
|
|
|
|
## .. code::
|
|
|
|
##
|
|
|
|
## nim c -r test fast_suite::mytest1 fast_suite::mytest2
|
|
|
|
## nim c -r test "fast_suite::mytest*"
|
|
|
|
## nim c -r test "auth*::" "crypto::hashing*"
|
|
|
|
## # Run suites starting with 'bug #' and standalone tests starting with '#'
|
|
|
|
## nim c -r test 'bug #*::' '::#*'
|
|
|
|
##
|
2021-04-29 12:20:03 +00:00
|
|
|
## Command line arguments
|
|
|
|
## ======================
|
|
|
|
##
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
## The unit test runner recognises serveral parameters that can be specified
|
|
|
|
## either via environment or command line, the latter taking precedence.
|
|
|
|
##
|
|
|
|
## Several options also have defaults that can be controlled at compile-time.
|
|
|
|
##
|
|
|
|
## --help Print short help and quit
|
|
|
|
## --xml:file Write JUnit-compatible XML report to `file`
|
|
|
|
## --console Write report to the console (default, when no other output
|
|
|
|
## is selected)
|
|
|
|
## --output-lvl:level Verbosity of output [COMPACT, VERBOSE, FAILURES, NONE] (env: UNITTEST2_OUTPUT_LVL)
|
|
|
|
## --verbose, -v Shorthand for --output-lvl:VERBOSE
|
2021-04-29 12:20:03 +00:00
|
|
|
##
|
|
|
|
## Command line parsing can be disabled with `-d:unittest2DisableParamFiltering`.
|
|
|
|
##
|
2019-05-19 21:11:17 +00:00
|
|
|
## Running tests in parallel
|
|
|
|
## =========================
|
|
|
|
##
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
## Early versions of this library had rudimentary support for running tests in
|
|
|
|
## parallel - this has since been removed due to safety issues in the
|
|
|
|
## implementation and may be reintroduced at a future date.
|
2019-05-29 14:33:25 +00:00
|
|
|
##
|
2019-05-19 21:11:17 +00:00
|
|
|
## Example
|
|
|
|
## -------
|
|
|
|
##
|
|
|
|
## .. code:: nim
|
|
|
|
##
|
|
|
|
## suite "description for this stuff":
|
|
|
|
## echo "suite setup: run once before the tests"
|
|
|
|
##
|
|
|
|
## setup:
|
|
|
|
## echo "run before each test"
|
|
|
|
##
|
|
|
|
## teardown:
|
|
|
|
## echo "run after each test"
|
|
|
|
##
|
|
|
|
## test "essential truths":
|
|
|
|
## # give up and stop if this fails
|
|
|
|
## require(true)
|
|
|
|
##
|
|
|
|
## test "slightly less obvious stuff":
|
|
|
|
## # print a nasty message and move on, skipping
|
|
|
|
## # the remainder of this block
|
|
|
|
## check(1 != 1)
|
|
|
|
## check("asd"[2] == 'd')
|
|
|
|
##
|
|
|
|
## test "out of bounds error is thrown on bad access":
|
|
|
|
## let v = @[1, 2, 3] # you can do initialization here
|
|
|
|
## expect(IndexError):
|
|
|
|
## discard v[4]
|
|
|
|
##
|
2019-05-28 21:58:13 +00:00
|
|
|
## suiteTeardown:
|
|
|
|
## echo "suite teardown: run once after the tests"
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
import std/[
|
|
|
|
macros, sequtils, sets, strutils, streams, tables, times, monotimes]
|
|
|
|
|
|
|
|
when defined(nimHasWarnBareExcept):
|
|
|
|
# In unit tests, we want to at least attempt to catch Exception no matter its
|
|
|
|
# UB
|
|
|
|
{.warning[BareExcept]: off.}
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
{.warning[LockLevel]: off.}
|
2023-02-14 03:44:13 +00:00
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
when declared(stdout):
|
2021-03-19 17:27:25 +00:00
|
|
|
import std/os
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
const useTerminal = declared(stdout) and not defined(js)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
type
|
|
|
|
OutputLevel* = enum ## The output verbosity of the tests.
|
|
|
|
VERBOSE, ## Print as much as possible.
|
|
|
|
COMPACT ## Print failures and compact success information
|
|
|
|
FAILURES, ## Print only failures
|
|
|
|
NONE ## Print nothing.
|
|
|
|
|
|
|
|
const
|
|
|
|
outputLevelDefault = COMPACT
|
|
|
|
slowThreshold = initDuration(seconds = 5)
|
|
|
|
|
|
|
|
# `unittest` compatibility
|
|
|
|
nimUnittestOutputLevel {.strdefine.} = $outputLevelDefault
|
|
|
|
nimUnittestColor {.strdefine.} = "auto" ## auto|on|off
|
|
|
|
nimUnittestAbortOnError {.booldefine.} = false
|
|
|
|
|
|
|
|
# `unittest2` compile-time configuration options
|
|
|
|
unittest2DisableParamFiltering {.booldefine.} = false
|
|
|
|
## Disables automatic command line argument parsing - parsing is available
|
|
|
|
## via the `parseParameters` function instead
|
|
|
|
unittest2Compat {.booldefine.} = true # This will be disabled in the future
|
|
|
|
## Compatibility mode for `unittest` for easier porting and improved
|
|
|
|
## backwards compatibility - no stability guarantees
|
|
|
|
unittest2NoCollect {.booldefine.} = false
|
|
|
|
## Disable test collection mode where tests are enumerated before they are
|
|
|
|
## run - in particular, this affects the order in which tests and suites
|
|
|
|
## have their bodies evaluated and disables several features that require
|
|
|
|
## knowing how many tests will be executed - experimental feature
|
|
|
|
unittest2PreviewIsolate {.booldefine.} = false
|
|
|
|
## Preview isolation mode where each test is run in a separate process - may
|
|
|
|
## be removed in the future
|
2021-04-28 08:19:23 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
when useTerminal:
|
|
|
|
import std/terminal
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
const
|
|
|
|
collect = (not unittest2NoCollect and not unittest2Compat) or unittest2PreviewIsolate
|
|
|
|
autoParseArgs = not unittest2DisableParamFiltering
|
|
|
|
isolate = unittest2PreviewIsolate
|
|
|
|
|
|
|
|
when isolate:
|
|
|
|
let
|
|
|
|
isolated = getEnv("UNITTEST2_ISOLATED") == "1"
|
|
|
|
## Test is running in the isolated environment
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2023-07-28 03:53:13 +00:00
|
|
|
from std/exitprocs import nil
|
|
|
|
template addExitProc(p: proc) =
|
|
|
|
try:
|
|
|
|
exitprocs.addExitProc(p)
|
|
|
|
except Exception as e:
|
|
|
|
echo "Can't add exit proc", e.msg
|
|
|
|
quit(1)
|
2023-02-14 02:14:47 +00:00
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
type
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
Test = object
|
|
|
|
suiteName: string
|
|
|
|
testName: string
|
|
|
|
impl: proc(suite, name: string): TestStatus
|
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
TestStatus* = enum ## The status of a test when it is done.
|
|
|
|
OK,
|
|
|
|
FAILED,
|
|
|
|
SKIPPED
|
|
|
|
|
|
|
|
TestResult* = object
|
|
|
|
suiteName*: string
|
|
|
|
## Name of the test suite that contains this test case.
|
|
|
|
testName*: string
|
|
|
|
## Name of the test case
|
|
|
|
status*: TestStatus
|
2021-04-27 14:34:39 +00:00
|
|
|
duration*: Duration # How long the test took, in seconds
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
output*: string
|
|
|
|
errors*: string
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
OutputFormatter* = ref object of RootObj
|
|
|
|
|
|
|
|
ConsoleOutputFormatter* = ref object of OutputFormatter
|
|
|
|
colorOutput: bool
|
|
|
|
## Have test results printed in color.
|
2021-03-19 17:27:25 +00:00
|
|
|
## Default is `auto` depending on `isatty(stdout)`, or override it with
|
|
|
|
## `-d:nimUnittestColor:auto|on|off`.
|
|
|
|
##
|
|
|
|
## Deprecated: Setting the environment variable `NIMTEST_COLOR` to `always`
|
|
|
|
## or `never` changes the default for the non-js target to true or false respectively.
|
|
|
|
## Deprecated: the environment variable `NIMTEST_NO_COLOR`, when set, changes the
|
|
|
|
## default to true, if `NIMTEST_COLOR` is undefined.
|
2019-05-19 21:11:17 +00:00
|
|
|
outputLevel: OutputLevel
|
|
|
|
## Set the verbosity of test results.
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
## Default is `VERBOSE`, or override with:
|
|
|
|
## `-d:nimUnittestOutputLevel:VERBOSE|FAILURES|NONE`.
|
2021-03-19 17:27:25 +00:00
|
|
|
##
|
|
|
|
## Deprecated: the `NIMTEST_OUTPUT_LVL` environment variable is set for the non-js target.
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
|
|
|
|
when collect:
|
|
|
|
tests: Table[string, int]
|
|
|
|
|
|
|
|
curSuiteName: string
|
|
|
|
curSuite: int
|
|
|
|
curTestName: string
|
|
|
|
curTest: int
|
|
|
|
|
|
|
|
statuses: array[TestStatus, int]
|
|
|
|
|
|
|
|
results: seq[TestResult]
|
|
|
|
|
|
|
|
failures: seq[TestResult]
|
|
|
|
|
|
|
|
errors: string
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-04-27 14:34:39 +00:00
|
|
|
JUnitTest = object
|
|
|
|
name: string
|
|
|
|
result: TestResult
|
|
|
|
error: (seq[string], string)
|
|
|
|
failures: seq[seq[string]]
|
|
|
|
|
|
|
|
JUnitSuite = object
|
|
|
|
name: string
|
|
|
|
tests: seq[JUnitTest]
|
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
JUnitOutputFormatter* = ref object of OutputFormatter
|
|
|
|
stream: Stream
|
2021-04-27 14:34:39 +00:00
|
|
|
defaultSuite: JUnitSuite
|
|
|
|
suites: seq[JUnitSuite]
|
|
|
|
currentSuite: int
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
# TODO these globals are threadvar so as to avoid gc-safety-issues - this should
|
|
|
|
# probably be resolved in a better way down the line specially since we
|
|
|
|
# don't support threads _really_
|
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
var
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
abortOnError* {.threadvar.}: bool
|
|
|
|
## Set to true in order to quit
|
|
|
|
## immediately on fail. Default is false,
|
|
|
|
## or override with `-d:nimUnittestAbortOnError:on|off`.
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
checkpoints {.threadvar.}: seq[string]
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatters {.threadvar.}: seq[OutputFormatter]
|
|
|
|
testsFilters {.threadvar.}: HashSet[string]
|
2021-03-19 17:27:25 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
when collect:
|
|
|
|
var
|
|
|
|
tests: OrderedTable[string, seq[Test]]
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
abortOnError = nimUnittestAbortOnError
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
when declared(stdout):
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
if existsEnv("UNITTEST2_ABORT_ON_ERROR") or existsEnv("NIMTEST_ABORT_ON_ERROR"):
|
2021-03-19 17:27:25 +00:00
|
|
|
abortOnError = true
|
2019-05-28 19:41:04 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
when collect:
|
|
|
|
method suiteRunStarted*(
|
|
|
|
formatter: OutputFormatter, tests: OrderedTable[string, seq[Test]]) {.base, gcsafe.} =
|
|
|
|
# Run when a round of running discovered suites starts - these may result
|
|
|
|
# in subsequent tests being added meaning subsequent suite runs
|
|
|
|
discard
|
2019-05-19 21:11:17 +00:00
|
|
|
method suiteStarted*(formatter: OutputFormatter, suiteName: string) {.base, gcsafe.} =
|
|
|
|
discard
|
|
|
|
method testStarted*(formatter: OutputFormatter, testName: string) {.base, gcsafe.} =
|
|
|
|
discard
|
2021-03-19 17:27:25 +00:00
|
|
|
method failureOccurred*(formatter: OutputFormatter, checkpoints: seq[string],
|
2023-02-14 06:13:26 +00:00
|
|
|
stackTrace: string) {.base, gcsafe.} =
|
2019-05-19 21:11:17 +00:00
|
|
|
## ``stackTrace`` is provided only if the failure occurred due to an exception.
|
|
|
|
## ``checkpoints`` is never ``nil``.
|
|
|
|
discard
|
|
|
|
method testEnded*(formatter: OutputFormatter, testResult: TestResult) {.base, gcsafe.} =
|
|
|
|
discard
|
|
|
|
method suiteEnded*(formatter: OutputFormatter) {.base, gcsafe.} =
|
|
|
|
discard
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
when collect:
|
|
|
|
method suiteRunEnded*(
|
|
|
|
formatter: OutputFormatter) {.base, gcsafe.} =
|
|
|
|
discard
|
|
|
|
|
2021-04-27 14:34:39 +00:00
|
|
|
method testRunEnded*(formatter: OutputFormatter) {.base, gcsafe.} =
|
|
|
|
# Runs when the test executable is about to end, which is implemented using
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
# addExitProc, a best-effort kind of place to do cleanups
|
2021-04-27 14:34:39 +00:00
|
|
|
discard
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
when collect:
|
|
|
|
proc suiteRunStarted(tests: OrderedTable[string, seq[Test]]) =
|
|
|
|
for formatter in formatters:
|
|
|
|
formatter.suiteRunStarted(tests)
|
|
|
|
|
|
|
|
proc suiteStarted(name: string) =
|
|
|
|
for formatter in formatters:
|
|
|
|
formatter.suiteStarted(name)
|
|
|
|
|
|
|
|
proc testStarted(name: string) =
|
|
|
|
for formatter in formatters:
|
|
|
|
formatter.testStarted(name)
|
|
|
|
|
|
|
|
proc testEnded(testResult: TestResult) =
|
|
|
|
for formatter in formatters:
|
|
|
|
formatter.testEnded(testResult)
|
|
|
|
|
|
|
|
proc suiteEnded() =
|
|
|
|
for formatter in formatters:
|
|
|
|
formatter.suiteEnded()
|
|
|
|
|
|
|
|
when collect:
|
|
|
|
proc suiteRunEnded() =
|
|
|
|
for formatter in formatters:
|
|
|
|
formatter.suiteRunEnded()
|
|
|
|
|
|
|
|
proc testRunEnded() =
|
|
|
|
for formatter in formatters:
|
|
|
|
testRunEnded(formatter)
|
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
proc addOutputFormatter*(formatter: OutputFormatter) =
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatters.add(formatter)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
proc resetOutputFormatters*() =
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatters.reset()
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
proc newConsoleOutputFormatter*(outputLevel: OutputLevel = outputLevelDefault,
|
2019-05-19 21:11:17 +00:00
|
|
|
colorOutput = true): ConsoleOutputFormatter =
|
|
|
|
ConsoleOutputFormatter(
|
|
|
|
outputLevel: outputLevel,
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
colorOutput: colorOutput,
|
2019-05-19 21:11:17 +00:00
|
|
|
)
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc defaultColorOutput(): bool =
|
2021-03-19 17:27:25 +00:00
|
|
|
let color = nimUnittestColor
|
|
|
|
case color
|
|
|
|
of "auto":
|
|
|
|
when declared(stdout): result = isatty(stdout)
|
|
|
|
else: result = false
|
|
|
|
of "on": result = true
|
|
|
|
of "off": result = false
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
else: raiseAssert "Unrecognised nimUnittestColor setting: " & color
|
2021-03-19 17:27:25 +00:00
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
when declared(stdout):
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
# TODO unittest2-equivalent color parsing
|
2019-05-19 21:11:17 +00:00
|
|
|
if existsEnv("NIMTEST_COLOR"):
|
2021-03-19 17:27:25 +00:00
|
|
|
let colorEnv = getEnv("NIMTEST_COLOR")
|
2019-05-19 21:11:17 +00:00
|
|
|
if colorEnv == "never":
|
2021-03-19 17:27:25 +00:00
|
|
|
result = false
|
2019-05-19 21:11:17 +00:00
|
|
|
elif colorEnv == "always":
|
2021-03-19 17:27:25 +00:00
|
|
|
result = true
|
2019-05-19 21:11:17 +00:00
|
|
|
elif existsEnv("NIMTEST_NO_COLOR"):
|
2021-03-19 17:27:25 +00:00
|
|
|
result = false
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc defaultOutputLevel(): OutputLevel =
|
2021-03-19 17:27:25 +00:00
|
|
|
when declared(stdout):
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
const levelEnv = "UNITTEST2_OUTPUT_LVL"
|
|
|
|
const nimtestEnv = "NIMTEST_OUTPUT_LVL"
|
|
|
|
if existsEnv(levelEnv):
|
2021-03-19 17:27:25 +00:00
|
|
|
try:
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
parseEnum[OutputLevel](getEnv(levelEnv))
|
|
|
|
except ValueError:
|
|
|
|
echo "Cannot parse UNITTEST2_OUTPUT_LVL: ", getEnv(levelEnv)
|
2021-03-19 17:27:25 +00:00
|
|
|
quit 1
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
elif existsEnv(nimtestEnv):
|
|
|
|
# std-compatible parsing and translation
|
|
|
|
case toUpper(getEnv(nimtestEnv))
|
|
|
|
of "PRINT_ALL": OutputLevel.VERBOSE
|
|
|
|
of "PRINT_FAILURES": OutputLevel.FAILURES
|
|
|
|
of "PRINT_NONE": OutputLevel.NONE
|
|
|
|
else:
|
|
|
|
echo "Cannot parse NIMTEST_OUTPUT_LVL: ", getEnv(nimtestEnv)
|
|
|
|
quit 1
|
|
|
|
else:
|
|
|
|
const defaultLevel = static: nimUnittestOutputLevel.parseEnum[:OutputLevel]
|
|
|
|
defaultLevel
|
|
|
|
|
|
|
|
proc defaultConsoleFormatter*(): ConsoleOutputFormatter =
|
|
|
|
newConsoleOutputFormatter(defaultOutputLevel(), defaultColorOutput())
|
2021-03-19 17:27:25 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
const
|
|
|
|
maxStatusLen = 7
|
|
|
|
maxDurationLen = 6
|
|
|
|
|
|
|
|
when collect:
|
|
|
|
proc formatFraction(cur, total: int): string =
|
|
|
|
let
|
|
|
|
cur = $cur
|
|
|
|
total = $total
|
|
|
|
"[" & align(cur, max(0, maxStatusLen - total.len - 1)) & "/" & total & "]"
|
|
|
|
|
|
|
|
template write(
|
|
|
|
formatter: ConsoleOutputFormatter, styled: untyped, unstyled: untyped) =
|
|
|
|
template ignoreExceptions(body: untyped) =
|
|
|
|
# We ignore exceptions throughout assuming there's no way to
|
|
|
|
try: body except CatchableError: discard
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
when useTerminal:
|
2019-05-19 21:11:17 +00:00
|
|
|
if formatter.colorOutput:
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
ignoreExceptions: styled
|
|
|
|
else: ignoreExceptions: unstyled
|
|
|
|
else: ignoreExceptions: unstyled
|
|
|
|
|
|
|
|
when collect:
|
|
|
|
method suiteRunStarted*(
|
|
|
|
formatter: ConsoleOutputFormatter, tests: OrderedTable[string, seq[Test]]) =
|
|
|
|
for k, v in tests:
|
|
|
|
formatter.tests[k] = v.len
|
|
|
|
|
|
|
|
when collect:
|
|
|
|
method suiteRunEnded*(formatter: ConsoleOutputFormatter) =
|
|
|
|
formatter.tests.reset()
|
|
|
|
|
|
|
|
method suiteStarted*(formatter: ConsoleOutputFormatter, suiteName: string) =
|
|
|
|
formatter.curSuiteName = suiteName
|
|
|
|
formatter.curSuite += 1
|
|
|
|
|
|
|
|
formatter.curTest.reset()
|
|
|
|
|
|
|
|
if formatter.outputLevel in {OutputLevel.FAILURES, OutputLevel.NONE}:
|
|
|
|
return
|
|
|
|
|
|
|
|
let
|
|
|
|
counter =
|
|
|
|
when collect: formatFraction(formatter.curSuite, formatter.tests.len) & " "
|
|
|
|
else:
|
|
|
|
if formatter.outputLevel == VERBOSE: "[Suite ] " else: ""
|
|
|
|
maxNameLen = when collect: max(toSeq(formatter.tests.keys()).mapIt(it.len)) else: 0
|
|
|
|
eol = if formatter.outputLevel == VERBOSE: "\n" else: " "
|
|
|
|
formatter.write do:
|
|
|
|
stdout.styledWrite(styleBright, fgBlue, counter, alignLeft(suiteName, maxNameLen), eol)
|
|
|
|
do:
|
|
|
|
stdout.write(counter, alignLeft(suiteName, maxNameLen), eol)
|
|
|
|
|
|
|
|
proc writeTestName(formatter: ConsoleOutputFormatter, testName: string) =
|
|
|
|
formatter.write do:
|
|
|
|
stdout.styledWrite fgBlue, testName
|
|
|
|
do:
|
|
|
|
stdout.write(testName)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
method testStarted*(formatter: ConsoleOutputFormatter, testName: string) =
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatter.curTestName = testName
|
|
|
|
formatter.curTest += 1
|
|
|
|
|
|
|
|
if formatter.outputLevel != VERBOSE:
|
|
|
|
return
|
|
|
|
|
|
|
|
# In verbose mode, print a line when the test starts so that output can be
|
|
|
|
# correlated with the test that's currently running rather than misleadingly
|
|
|
|
# being printed just below the test that just finished running.
|
|
|
|
let
|
|
|
|
counter =
|
|
|
|
when collect:
|
|
|
|
try: formatFraction(formatter.curTest, formatter.tests[formatter.curSuiteName]) & " "
|
|
|
|
except CatchableError: ""
|
|
|
|
else:
|
|
|
|
"[Test ]"
|
|
|
|
|
|
|
|
formatter.write do:
|
|
|
|
stdout.styledWrite " ", fgBlue, alignLeft(counter, maxStatusLen + maxDurationLen + 7)
|
|
|
|
do:
|
|
|
|
stdout.write " ", alignLeft(counter, maxStatusLen + maxDurationLen + 7)
|
|
|
|
|
|
|
|
writeTestName(formatter, testName)
|
|
|
|
echo ""
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
method failureOccurred*(formatter: ConsoleOutputFormatter,
|
|
|
|
checkpoints: seq[string], stackTrace: string) =
|
2019-05-19 21:11:17 +00:00
|
|
|
if stackTrace.len > 0:
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatter.errors.add(stackTrace)
|
|
|
|
formatter.errors.add("\n")
|
2019-05-19 21:11:17 +00:00
|
|
|
for msg in items(checkpoints):
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatter.errors.add(" ")
|
|
|
|
formatter.errors.add(msg)
|
|
|
|
formatter.errors.add("\n")
|
|
|
|
|
|
|
|
proc color(status: TestStatus): ForegroundColor =
|
|
|
|
case status
|
|
|
|
of TestStatus.OK: fgGreen
|
|
|
|
of TestStatus.FAILED: fgRed
|
|
|
|
of TestStatus.SKIPPED: fgYellow
|
|
|
|
proc marker(status: TestStatus): string =
|
|
|
|
case status
|
|
|
|
of TestStatus.OK: "."
|
|
|
|
of TestStatus.FAILED: "F"
|
|
|
|
of TestStatus.SKIPPED: "s"
|
|
|
|
|
|
|
|
proc formatDuration(dur: Duration, aligned = true): string =
|
|
|
|
let
|
|
|
|
seconds = dur.inMilliseconds.float / 1000.0
|
|
|
|
precision = max(3 - ($seconds.int).len, 1)
|
|
|
|
str = formatFloat(seconds, ffDecimal, precision)
|
|
|
|
|
|
|
|
if aligned:
|
|
|
|
"(" & align(str, maxDurationLen) & "s)"
|
|
|
|
else:
|
|
|
|
"(" & str & "s)"
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc formatStatus(status: TestStatus): string =
|
|
|
|
"[" & alignLeft($status, maxStatusLen) & "]"
|
|
|
|
|
|
|
|
proc getAppFilename2(): string =
|
|
|
|
# TODO https://github.com/nim-lang/Nim/pull/22544
|
|
|
|
try:
|
|
|
|
getAppFilename()
|
|
|
|
except OSError:
|
|
|
|
""
|
|
|
|
|
|
|
|
proc printFailureInfo(formatter: ConsoleOutputFormatter, testResult: TestResult) =
|
|
|
|
# Show how to re-run this test case
|
|
|
|
echo repeat('=', testResult.testName.len)
|
|
|
|
echo " ", getAppFilename2(), " ", quoteShell(testResult.suiteName & "::" & testResult.testName)
|
|
|
|
echo repeat('-', testResult.testName.len)
|
|
|
|
|
|
|
|
# Show the output
|
|
|
|
if testResult.output.len > 0:
|
|
|
|
echo testResult.output
|
|
|
|
if testResult.errors.len > 0:
|
|
|
|
echo testResult.errors
|
|
|
|
|
|
|
|
proc printTestResultStatus(formatter: ConsoleOutputFormatter, testResult: TestResult) =
|
|
|
|
let
|
|
|
|
status = formatStatus(testResult.status)
|
|
|
|
duration = formatDuration(testResult.duration)
|
|
|
|
|
|
|
|
formatter.write do:
|
|
|
|
stdout.styledWrite(
|
|
|
|
" ", styleBright, testResult.status.color, status, " ")
|
|
|
|
if testResult.duration > slowThreshold:
|
|
|
|
stdout.styledWrite styleBright, duration
|
|
|
|
else:
|
|
|
|
stdout.write(duration)
|
|
|
|
stdout.write " ", testResult.testName
|
|
|
|
do:
|
|
|
|
stdout.styledWrite " ", status, " ", duration, " ", testResult.testName
|
|
|
|
echo ""
|
2022-05-31 15:12:44 +00:00
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
method testEnded*(formatter: ConsoleOutputFormatter, testResult: TestResult) =
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
formatter.statuses[testResult.status] += 1
|
|
|
|
|
|
|
|
if formatter.outputLevel == NONE:
|
|
|
|
return
|
|
|
|
|
|
|
|
var testResult = testResult
|
|
|
|
testResult.errors = move(formatter.errors)
|
|
|
|
|
|
|
|
formatter.results.add(testResult)
|
|
|
|
|
|
|
|
if formatter.outputLevel == VERBOSE and testResult.status == TestStatus.FAILED:
|
|
|
|
# We'll print it again when all tests have completed
|
|
|
|
formatter.failures.add testResult
|
|
|
|
|
|
|
|
if formatter.outputLevel in {VERBOSE, FAILURES}:
|
|
|
|
if testResult.status == TestStatus.FAILED:
|
|
|
|
printFailureInfo(formatter, testResult)
|
|
|
|
if formatter.outputLevel == VERBOSE or testResult.status == TestStatus.FAILED:
|
|
|
|
printTestResultStatus(formatter, testResult)
|
|
|
|
else:
|
|
|
|
# In compact mode, we use a small marker to mark progress within the suite -
|
|
|
|
# we have to be careful about line breaks and flushing so that the marker
|
|
|
|
# really ends up on the screen where it's supposed to
|
|
|
|
# TODO if the test writes to stdout, the display with be disrupted
|
|
|
|
# capturing / redirecting stdout with `dup2` or process isolation could
|
|
|
|
# fix this
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2022-05-31 15:12:44 +00:00
|
|
|
let
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
marker = testResult.status.marker()
|
|
|
|
color = testResult.status.color()
|
|
|
|
formatter.write do:
|
|
|
|
stdout.styledWrite styleBright, color, marker
|
|
|
|
do:
|
|
|
|
stdout.write marker
|
|
|
|
stdout.flushFile()
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
method suiteEnded*(formatter: ConsoleOutputFormatter) =
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
if formatter.outputLevel == OutputLevel.NONE:
|
|
|
|
return
|
|
|
|
|
|
|
|
let
|
|
|
|
totalDur = formatter.results.foldl(a + b.duration, DurationZero)
|
|
|
|
totalDurStr = formatDuration(totalDur, false)
|
|
|
|
|
|
|
|
if formatter.outputLevel == OutputLevel.COMPACT:
|
|
|
|
# Complete the line with timing information
|
|
|
|
formatter.write do:
|
|
|
|
if totalDur > slowThreshold:
|
|
|
|
stdout.styledWrite(" ", styleBright, totalDurStr)
|
|
|
|
else:
|
|
|
|
stdout.write(" ", totalDurStr)
|
|
|
|
echo ""
|
|
|
|
do:
|
|
|
|
echo(" ", totalDurStr)
|
|
|
|
|
|
|
|
var failed = false
|
|
|
|
if formatter.outputLevel notin {VERBOSE, FAILURES}:
|
|
|
|
for testResult in formatter.results:
|
|
|
|
if testResult.status == TestStatus.FAILED:
|
|
|
|
failed = true
|
|
|
|
formatter.printFailureInfo(testResult)
|
|
|
|
formatter.printTestResultStatus(testResult)
|
|
|
|
echo ""
|
|
|
|
|
|
|
|
formatter.results.reset()
|
|
|
|
|
|
|
|
if failed or formatter.outputLevel == VERBOSE:
|
|
|
|
formatter.write do:
|
|
|
|
if totalDur > slowThreshold:
|
|
|
|
stdout.styledWrite styleBright, align(totalDurStr, maxStatusLen)
|
|
|
|
else:
|
|
|
|
stdout.write(align(totalDurStr, maxStatusLen))
|
|
|
|
do:
|
|
|
|
stdout.write(align(totalDurStr, maxStatusLen))
|
|
|
|
|
|
|
|
echo(" ", formatter.curSuiteName)
|
|
|
|
echo("")
|
|
|
|
|
|
|
|
method testRunEnded*(formatter: ConsoleOutputFormatter) =
|
|
|
|
if formatter.outputLevel notin {VERBOSE, COMPACT} or
|
|
|
|
(formatter.outputLevel == FAILURES and
|
|
|
|
formatter.statuses[TestStatus.FAILED] > 0):
|
|
|
|
return
|
|
|
|
|
|
|
|
try:
|
|
|
|
let total = foldl(formatter.statuses, a + b, 0)
|
|
|
|
stdout.write("[Summary ] ", $total, " tests run: ")
|
|
|
|
|
|
|
|
var first = true
|
|
|
|
for s, c in formatter.statuses:
|
|
|
|
if first:
|
|
|
|
first = false
|
|
|
|
else:
|
|
|
|
stdout.write(", ")
|
|
|
|
if c > 0:
|
|
|
|
formatter.write do: stdout.styledWrite(s.color, $c, " ", $s)
|
|
|
|
do: stdout.write($c, " ", $s)
|
|
|
|
else:
|
|
|
|
stdout.write($c, " ", $s)
|
|
|
|
echo ""
|
|
|
|
except CatchableError: discard
|
|
|
|
|
|
|
|
# In verbose mode, it's likely failures got spammed away - print the specifics
|
|
|
|
# so that they can more easily be looked up:
|
|
|
|
for testResult in formatter.failures:
|
|
|
|
formatter.printTestResultStatus(testResult)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
proc xmlEscape(s: string): string =
|
|
|
|
result = newStringOfCap(s.len)
|
|
|
|
for c in items(s):
|
|
|
|
case c:
|
|
|
|
of '<': result.add("<")
|
|
|
|
of '>': result.add(">")
|
|
|
|
of '&': result.add("&")
|
|
|
|
of '"': result.add(""")
|
|
|
|
of '\'': result.add("'")
|
|
|
|
else:
|
|
|
|
if ord(c) < 32:
|
|
|
|
result.add("&#" & $ord(c) & ';')
|
|
|
|
else:
|
|
|
|
result.add(c)
|
|
|
|
|
|
|
|
proc newJUnitOutputFormatter*(stream: Stream): JUnitOutputFormatter =
|
|
|
|
## Creates a formatter that writes report to the specified stream in
|
|
|
|
## JUnit format.
|
|
|
|
## The ``stream`` is NOT closed automatically when the test are finished,
|
|
|
|
## because the formatter has no way to know when all tests are finished.
|
|
|
|
## You should invoke formatter.close() to finalize the report.
|
|
|
|
result = JUnitOutputFormatter(
|
|
|
|
stream: stream,
|
2021-04-27 14:34:39 +00:00
|
|
|
defaultSuite: JUnitSuite(name: "default"),
|
|
|
|
currentSuite: -1,
|
2019-05-19 21:11:17 +00:00
|
|
|
)
|
2021-03-19 17:27:25 +00:00
|
|
|
try:
|
|
|
|
stream.writeLine("<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
|
|
|
|
except CatchableError as exc:
|
|
|
|
echo "Cannot write JUnit: ", exc.msg
|
|
|
|
quit 1
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-04-27 14:34:39 +00:00
|
|
|
template suite(formatter: JUnitOutputFormatter): untyped =
|
|
|
|
if formatter.currentSuite == -1:
|
|
|
|
addr formatter.defaultSuite
|
|
|
|
else:
|
|
|
|
addr formatter.suites[formatter.currentSuite]
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
method suiteStarted*(formatter: JUnitOutputFormatter, suiteName: string) =
|
2021-04-27 14:34:39 +00:00
|
|
|
formatter.currentSuite = formatter.suites.len()
|
|
|
|
formatter.suites.add(JUnitSuite(name: suiteName))
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
method testStarted*(formatter: JUnitOutputFormatter, testName: string) =
|
2021-04-27 14:34:39 +00:00
|
|
|
formatter.suite().tests.add(JUnitTest(name: testName))
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
method failureOccurred*(formatter: JUnitOutputFormatter,
|
|
|
|
checkpoints: seq[string], stackTrace: string) =
|
2019-05-19 21:11:17 +00:00
|
|
|
## ``stackTrace`` is provided only if the failure occurred due to an exception.
|
|
|
|
## ``checkpoints`` is never ``nil``.
|
|
|
|
if stackTrace.len > 0:
|
2021-04-27 14:34:39 +00:00
|
|
|
formatter.suite().tests[^1].error = (checkpoints, stackTrace)
|
|
|
|
else:
|
|
|
|
formatter.suite().tests[^1].failures.add(checkpoints)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
method testEnded*(formatter: JUnitOutputFormatter, testResult: TestResult) =
|
2021-04-27 14:34:39 +00:00
|
|
|
formatter.suite().tests[^1].result = testResult
|
|
|
|
|
|
|
|
method suiteEnded*(formatter: JUnitOutputFormatter) =
|
|
|
|
formatter.currentSuite = -1
|
|
|
|
|
|
|
|
func toFloatSeconds(duration: Duration): float64 =
|
|
|
|
duration.inNanoseconds().float64 / 1_000_000_000.0
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc writeTest(s: Stream, test: JUnitTest) {.raises: [CatchableError].} =
|
2021-04-27 14:34:39 +00:00
|
|
|
let
|
|
|
|
time = test.result.duration.toFloatSeconds()
|
|
|
|
timeStr = time.formatFloat(ffDecimal, precision = 6)
|
|
|
|
|
|
|
|
s.writeLine("\t\t<testcase name=\"$#\" time=\"$#\">" % [
|
|
|
|
xmlEscape(test.name), timeStr])
|
|
|
|
case test.result.status
|
|
|
|
of TestStatus.OK:
|
|
|
|
discard
|
|
|
|
of TestStatus.SKIPPED:
|
|
|
|
s.writeLine("\t\t\t<skipped />")
|
|
|
|
of TestStatus.FAILED:
|
|
|
|
if test.error[0].len > 0:
|
|
|
|
s.writeLine("\t\t\t<error message=\"$#\">$#</error>" % [
|
|
|
|
xmlEscape(join(test.error[0], "\n")), xmlEscape(test.error[1])])
|
|
|
|
|
|
|
|
for failure in test.failures:
|
|
|
|
s.writeLine("\t\t\t<failure message=\"$#\">$#</failure>" %
|
|
|
|
[xmlEscape(failure[^1]), xmlEscape(join(failure[0..^2], "\n"))])
|
|
|
|
|
|
|
|
s.writeLine("\t\t</testcase>")
|
|
|
|
|
|
|
|
proc countTests(counts: var (int, int, int, int, float), suite: JUnitSuite) =
|
|
|
|
counts[0] += suite.tests.len()
|
|
|
|
for test in suite.tests:
|
|
|
|
counts[4] += test.result.duration.toFloatSeconds()
|
|
|
|
case test.result.status
|
2021-03-19 17:27:25 +00:00
|
|
|
of TestStatus.OK:
|
|
|
|
discard
|
|
|
|
of TestStatus.SKIPPED:
|
2021-04-27 14:34:39 +00:00
|
|
|
counts[3] += 1
|
2021-03-19 17:27:25 +00:00
|
|
|
of TestStatus.FAILED:
|
2021-04-27 14:34:39 +00:00
|
|
|
if test.error[0].len > 0:
|
|
|
|
counts[2] += 1
|
2021-03-19 17:27:25 +00:00
|
|
|
else:
|
2021-04-27 14:34:39 +00:00
|
|
|
counts[1] += 1
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc writeSuite(s: Stream, suite: JUnitSuite) {.raises: [CatchableError].} =
|
2021-04-27 14:34:39 +00:00
|
|
|
var counts: (int, int, int, int, float)
|
|
|
|
countTests(counts, suite)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-04-27 14:34:39 +00:00
|
|
|
let timeStr = counts[4].formatFloat(ffDecimal, precision = 6)
|
|
|
|
|
2021-04-27 15:41:56 +00:00
|
|
|
s.writeLine("\t" & """<testsuite name="$1" tests="$2" failures="$3" errors="$4" skipped="$5" time="$6">""" % [
|
2021-04-27 14:34:39 +00:00
|
|
|
xmlEscape(suite.name), $counts[0], $counts[1], $counts[2], $counts[3], timeStr])
|
|
|
|
|
|
|
|
for test in suite.tests.items():
|
|
|
|
s.writeTest(test)
|
|
|
|
|
|
|
|
s.writeLine("\t</testsuite>")
|
|
|
|
|
|
|
|
method testRunEnded*(formatter: JUnitOutputFormatter) =
|
|
|
|
## Completes the report and closes the underlying stream.
|
|
|
|
let s = formatter.stream
|
2023-01-04 14:48:30 +00:00
|
|
|
|
2023-02-17 14:23:16 +00:00
|
|
|
when defined(nimHasWarnBareExcept):
|
2023-01-04 14:48:30 +00:00
|
|
|
{.warning[BareExcept]:off.}
|
2021-03-19 17:27:25 +00:00
|
|
|
try:
|
2021-04-27 14:34:39 +00:00
|
|
|
s.writeLine("<testsuites>")
|
|
|
|
|
|
|
|
for suite in formatter.suites.mitems():
|
|
|
|
s.writeSuite(suite)
|
|
|
|
|
|
|
|
if formatter.defaultSuite.tests.len() > 0:
|
|
|
|
s.writeSuite(formatter.defaultSuite)
|
|
|
|
|
|
|
|
s.writeLine("</testsuites>")
|
|
|
|
s.close()
|
|
|
|
except Exception as exc: # Work around Exception raised in stream
|
2021-03-19 17:27:25 +00:00
|
|
|
echo "Cannot write JUnit: ", exc.msg
|
|
|
|
quit 1
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2023-02-17 14:23:16 +00:00
|
|
|
when defined(nimHasWarnBareExcept):
|
2023-01-04 14:48:30 +00:00
|
|
|
{.warning[BareExcept]:on.}
|
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
proc glob(matcher, filter: string): bool =
|
|
|
|
## Globbing using a single `*`. Empty `filter` matches everything.
|
|
|
|
if filter.len == 0:
|
|
|
|
return true
|
|
|
|
|
|
|
|
if not filter.contains('*'):
|
|
|
|
return matcher == filter
|
|
|
|
|
|
|
|
let beforeAndAfter = filter.split('*', maxsplit=1)
|
|
|
|
if beforeAndAfter.len == 1:
|
|
|
|
# "foo*"
|
2021-03-19 17:27:25 +00:00
|
|
|
return matcher.startsWith(beforeAndAfter[0])
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
if matcher.len < filter.len - 1:
|
|
|
|
return false # "12345" should not match "123*345"
|
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
return matcher.startsWith(beforeAndAfter[0]) and matcher.endsWith(
|
|
|
|
beforeAndAfter[1])
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
proc matchFilter(suiteName, testName, filter: string): bool =
|
|
|
|
if filter == "":
|
|
|
|
return true
|
|
|
|
if testName == filter:
|
|
|
|
# corner case for tests containing "::" in their name
|
|
|
|
return true
|
|
|
|
let suiteAndTestFilters = filter.split("::", maxsplit=1)
|
|
|
|
|
|
|
|
if suiteAndTestFilters.len == 1:
|
|
|
|
# no suite specified
|
2021-03-19 17:27:25 +00:00
|
|
|
let testFilter = suiteAndTestFilters[0]
|
|
|
|
return glob(testName, testFilter)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
return glob(suiteName, suiteAndTestFilters[0]) and
|
|
|
|
glob(testName, suiteAndTestFilters[1])
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
when defined(testing): export matchFilter
|
|
|
|
|
|
|
|
proc shouldRun(currentSuiteName, testName: string): bool =
|
|
|
|
## Check if a test should be run by matching suiteName and testName against
|
|
|
|
## test filters.
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
if testsFilters.len == 0:
|
|
|
|
return true
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
for f in testsFilters:
|
|
|
|
if matchFilter(currentSuiteName, testName, f):
|
|
|
|
return true
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
return false
|
|
|
|
|
2021-04-28 08:19:23 +00:00
|
|
|
proc parseParameters*(args: openArray[string]) =
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
var
|
|
|
|
hasConsole = false
|
|
|
|
hasXml: string
|
|
|
|
hasVerbose = false
|
|
|
|
hasLevel = defaultOutputLevel()
|
|
|
|
|
|
|
|
# Read tests to run from the command line.
|
|
|
|
for str in args:
|
|
|
|
if str.startsWith("--help"):
|
|
|
|
echo "Usage: [--xml=file.xml] [--console] [--output-level=[VERBOSE,COMPACT,FAILURES,NONE]] [test-name-glob]"
|
|
|
|
quit 0
|
|
|
|
elif str.startsWith("--xml:") or str.startsWith("--xml="):
|
|
|
|
hasXml = str[("--xml".len + 1)..^1] # skip separator char as well
|
|
|
|
elif str.startsWith("--console"):
|
|
|
|
hasConsole = true
|
|
|
|
elif str.startsWith("--output-level:") or str.startsWith("--output-level="):
|
|
|
|
hasLevel = try: parseEnum[OutputLevel](str[("--output-level".len + 1)..^1])
|
|
|
|
except ValueError:
|
|
|
|
echo "Unknown output level ", str[("--output-level".len + 1)..^1]
|
|
|
|
quit 1
|
|
|
|
elif str.startsWith("--verbose") or str == "-v":
|
|
|
|
hasVerbose = true
|
|
|
|
else:
|
|
|
|
testsFilters.incl(str)
|
|
|
|
if hasXml.len > 0:
|
|
|
|
try:
|
|
|
|
formatters.add(newJUnitOutputFormatter(newFileStream(hasXml, fmWrite)))
|
|
|
|
except CatchableError as exc:
|
|
|
|
echo "Cannot open ", hasXml, " for writing: ", exc.msg
|
|
|
|
quit 1
|
|
|
|
|
|
|
|
if hasConsole or hasXml.len == 0:
|
|
|
|
let level =
|
|
|
|
if hasVerbose: OutputLevel.VERBOSE
|
|
|
|
else: hasLevel
|
|
|
|
formatters.add(newConsoleOutputFormatter(level, defaultColorOutput()))
|
2021-04-28 08:19:23 +00:00
|
|
|
|
|
|
|
proc ensureInitialized() =
|
|
|
|
if autoParseArgs and declared(paramCount):
|
|
|
|
parseParameters(commandLineParams())
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
if formatters.len == 0:
|
|
|
|
formatters = @[OutputFormatter(defaultConsoleFormatter())]
|
2021-04-27 14:34:39 +00:00
|
|
|
|
|
|
|
ensureInitialized() # Run once!
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
template suite*(nameParam: string, body: untyped) {.dirty.} =
|
2019-05-19 21:11:17 +00:00
|
|
|
## Declare a test suite identified by `name` with optional ``setup``
|
|
|
|
## and/or ``teardown`` section.
|
|
|
|
##
|
|
|
|
## A test suite is a series of one or more related tests sharing a
|
|
|
|
## common fixture (``setup``, ``teardown``). The fixture is executed
|
|
|
|
## for EACH test.
|
|
|
|
##
|
|
|
|
## .. code-block:: nim
|
|
|
|
## suite "test suite for addition":
|
|
|
|
## setup:
|
|
|
|
## let result = 4
|
|
|
|
##
|
|
|
|
## test "2 + 2 = 4":
|
|
|
|
## check(2+2 == result)
|
|
|
|
##
|
|
|
|
## test "(2 + -2) != 4":
|
|
|
|
## check(2 + -2 != result)
|
|
|
|
##
|
|
|
|
## # No teardown needed
|
|
|
|
##
|
|
|
|
## The suite will run the individual test cases in the order in which
|
|
|
|
## they were listed. With default global settings the above code prints:
|
|
|
|
##
|
|
|
|
## .. code-block::
|
|
|
|
##
|
|
|
|
## [Suite] test suite for addition
|
|
|
|
## [OK] 2 + 2 = 4
|
|
|
|
## [OK] (2 + -2) != 4
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
bind collect, suiteStarted, suiteEnded
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
block:
|
|
|
|
template setup(setupBody: untyped) {.dirty, used.} =
|
|
|
|
var testSetupIMPLFlag {.used.} = true
|
|
|
|
template testSetupIMPL: untyped {.dirty.} = setupBody
|
|
|
|
|
|
|
|
template teardown(teardownBody: untyped) {.dirty, used.} =
|
|
|
|
var testTeardownIMPLFlag {.used.} = true
|
|
|
|
template testTeardownIMPL: untyped {.dirty.} = teardownBody
|
|
|
|
|
2019-05-28 21:58:13 +00:00
|
|
|
template suiteTeardown(suiteTeardownBody: untyped) {.dirty, used.} =
|
|
|
|
var testSuiteTeardownIMPLFlag {.used.} = true
|
|
|
|
template testSuiteTeardownIMPL: untyped {.dirty.} = suiteTeardownBody
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
let suiteName {.inject.} = nameParam
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
when not collect:
|
|
|
|
suiteStarted(suiteName)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
# TODO what about exceptions in the suite itself?
|
|
|
|
body
|
|
|
|
|
|
|
|
when declared(testSuiteTeardownIMPLFlag):
|
|
|
|
testSuiteTeardownIMPL()
|
|
|
|
|
|
|
|
when not collect:
|
|
|
|
suiteEnded()
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-08-16 13:10:04 +00:00
|
|
|
template checkpoint*(msg: string) =
|
|
|
|
## Set a checkpoint identified by `msg`. Upon test failure all
|
|
|
|
## checkpoints encountered so far are printed out. Example:
|
|
|
|
##
|
|
|
|
## .. code-block:: nim
|
|
|
|
##
|
|
|
|
## checkpoint("Checkpoint A")
|
|
|
|
## check((42, "the Answer to life and everything") == (1, "a"))
|
|
|
|
## checkpoint("Checkpoint B")
|
|
|
|
##
|
|
|
|
## outputs "Checkpoint A" once it fails.
|
|
|
|
checkpoints.add(msg)
|
|
|
|
# TODO: add support for something like SCOPED_TRACE from Google Test
|
|
|
|
|
|
|
|
template fail* =
|
|
|
|
## Print out the checkpoints encountered so far and quit if ``abortOnError``
|
|
|
|
## is true. Otherwise, erase the checkpoints and indicate the test has
|
|
|
|
## failed (change exit code and test status). This template is useful
|
|
|
|
## for debugging, but is otherwise mostly used internally. Example:
|
|
|
|
##
|
|
|
|
## .. code-block:: nim
|
|
|
|
##
|
|
|
|
## checkpoint("Checkpoint A")
|
|
|
|
## complicatedProcInThread()
|
|
|
|
## fail()
|
|
|
|
##
|
|
|
|
## outputs "Checkpoint A" before quitting.
|
|
|
|
when declared(testStatusIMPL):
|
|
|
|
testStatusIMPL = TestStatus.FAILED
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
|
|
|
|
programResult = 1
|
|
|
|
|
|
|
|
for formatter in formatters:
|
|
|
|
let formatter = formatter # avoid lent iterator
|
|
|
|
when declared(stackTrace):
|
|
|
|
when stackTrace is string:
|
|
|
|
formatter.failureOccurred(checkpoints, stackTrace)
|
|
|
|
else:
|
|
|
|
formatter.failureOccurred(checkpoints, "")
|
|
|
|
else:
|
|
|
|
formatter.failureOccurred(checkpoints, "")
|
2021-08-16 13:10:04 +00:00
|
|
|
|
|
|
|
if abortOnError: quit(1)
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
checkpoints.reset()
|
2021-08-16 13:10:04 +00:00
|
|
|
|
|
|
|
template skip* =
|
|
|
|
## Mark the test as skipped. Should be used directly
|
|
|
|
## in case when it is not possible to perform test
|
|
|
|
## for reasons depending on outer environment,
|
|
|
|
## or certain application logic conditions or configurations.
|
|
|
|
## The test code is still executed.
|
|
|
|
##
|
|
|
|
## .. code-block:: nim
|
|
|
|
##
|
|
|
|
## if not isGLContextCreated():
|
|
|
|
## skip()
|
|
|
|
bind checkpoints
|
|
|
|
|
|
|
|
testStatusIMPL = TestStatus.SKIPPED
|
|
|
|
checkpoints = @[]
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc runDirect(test: Test) =
|
|
|
|
let startTime = getMonoTime()
|
|
|
|
testStarted(test.testName)
|
|
|
|
|
|
|
|
# TODO this annotation works around a limitation where we know that we only
|
|
|
|
# call the callback from the main thread but the compiler doesn't -
|
|
|
|
# when / if testing becomes multithreaded, this will need a proper
|
|
|
|
# solution
|
|
|
|
{.gcsafe.}:
|
|
|
|
let
|
|
|
|
status = test.impl(test.suiteName, test.testName)
|
|
|
|
duration = getMonoTime() - startTime
|
|
|
|
|
|
|
|
testEnded(TestResult(
|
|
|
|
suiteName: test.suiteName,
|
|
|
|
testName: test.testName,
|
|
|
|
status: status,
|
|
|
|
duration: duration
|
|
|
|
))
|
|
|
|
|
|
|
|
template test*(nameParam: string, body: untyped) =
|
2019-05-19 21:11:17 +00:00
|
|
|
## Define a single test case identified by `name`.
|
|
|
|
##
|
|
|
|
## .. code-block:: nim
|
|
|
|
##
|
|
|
|
## test "roses are red":
|
|
|
|
## let roses = "red"
|
|
|
|
## check(roses == "red")
|
|
|
|
##
|
|
|
|
## The above code outputs:
|
|
|
|
##
|
|
|
|
## .. code-block::
|
|
|
|
##
|
|
|
|
## [OK] roses are red
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
bind collect, runDirect, shouldRun, checkpoints
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
proc runTest(suiteName, testName: string): TestStatus {.gensym.} =
|
2021-03-19 17:27:25 +00:00
|
|
|
var testStatusIMPL {.inject.} = TestStatus.OK
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
let suiteName {.inject, used.} = suiteName
|
|
|
|
let testName {.inject, used.} = testName
|
2021-04-27 14:34:39 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
try:
|
|
|
|
when declared(testSetupIMPLFlag): testSetupIMPL()
|
|
|
|
when declared(testTeardownIMPLFlag):
|
|
|
|
defer: testTeardownIMPL()
|
2021-04-27 15:41:56 +00:00
|
|
|
block:
|
|
|
|
body
|
2021-03-19 17:27:25 +00:00
|
|
|
|
2023-01-04 14:48:30 +00:00
|
|
|
except CatchableError as e:
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
let eTypeDesc = "[" & $e.name & "]"
|
|
|
|
checkpoint("Unhandled error: " & e.msg & " " & eTypeDesc)
|
|
|
|
var stackTrace {.inject.} = e.getStackTrace()
|
|
|
|
fail()
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2023-01-04 14:48:30 +00:00
|
|
|
except Defect as e: # This may or may not work dependings on --panics
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
let eTypeDesc = "[" & $e.name & "]"
|
2023-01-04 14:48:30 +00:00
|
|
|
checkpoint("Unhandled defect: " & e.msg & " " & eTypeDesc)
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
var stackTrace {.inject.} = e.getStackTrace()
|
|
|
|
fail()
|
|
|
|
except Exception as e:
|
|
|
|
let eTypeDesc = "[" & $e.name & "]"
|
|
|
|
checkpoint("Unhandled exception that may cause undefined behavior: " & e.msg & " " & eTypeDesc)
|
|
|
|
var stackTrace {.inject.} = e.getStackTrace()
|
|
|
|
fail()
|
2023-01-04 14:48:30 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
checkpoints = @[]
|
|
|
|
|
|
|
|
testStatusIMPL
|
|
|
|
|
|
|
|
let
|
|
|
|
localSuiteName =
|
|
|
|
when declared(suiteName):
|
|
|
|
suiteName
|
|
|
|
else: instantiationInfo().filename
|
|
|
|
localTestName = nameParam
|
|
|
|
if shouldRun(localSuiteName, localTestName):
|
|
|
|
let
|
|
|
|
instance =
|
|
|
|
Test(testName: localTestName, suiteName: localSuiteName, impl: runTest)
|
|
|
|
when collect:
|
|
|
|
tests.mgetOrPut(localSuiteName, default(seq[Test])).add(instance)
|
2021-03-19 17:27:25 +00:00
|
|
|
else:
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
runDirect(instance)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
{.pop.} # raises: []
|
2021-03-19 17:27:25 +00:00
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
macro check*(conditions: untyped): untyped =
|
|
|
|
## Verify if a statement or a list of statements is true.
|
|
|
|
## A helpful error message and set checkpoints are printed out on
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
## failure (if ``outputLevel`` is not ``NONE``).
|
2021-03-19 17:27:25 +00:00
|
|
|
runnableExamples:
|
|
|
|
import std/strutils
|
|
|
|
|
|
|
|
check("AKB48".toLowerAscii() == "akb48")
|
|
|
|
|
|
|
|
let teams = {'A', 'K', 'B', '4', '8'}
|
|
|
|
|
|
|
|
check:
|
|
|
|
"AKB48".toLowerAscii() == "akb48"
|
|
|
|
'C' notin teams
|
|
|
|
|
2023-01-04 14:48:30 +00:00
|
|
|
{.warning[Deprecated]:off.}
|
2019-05-19 21:11:17 +00:00
|
|
|
let checked = callsite()[1]
|
2023-01-04 14:48:30 +00:00
|
|
|
{.warning[Deprecated]:on.}
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
template asgn(a: untyped, value: typed) =
|
|
|
|
var a = value # XXX: we need "var: var" here in order to
|
|
|
|
# preserve the semantics of var params
|
|
|
|
|
|
|
|
template print(name: untyped, value: typed) =
|
|
|
|
when compiles(string($value)):
|
|
|
|
checkpoint(name & " was " & $value)
|
|
|
|
|
|
|
|
proc inspectArgs(exp: NimNode): tuple[assigns, check, printOuts: NimNode] =
|
|
|
|
result.check = copyNimTree(exp)
|
|
|
|
result.assigns = newNimNode(nnkStmtList)
|
|
|
|
result.printOuts = newNimNode(nnkStmtList)
|
|
|
|
|
|
|
|
var counter = 0
|
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
if exp[0].kind in {nnkIdent, nnkOpenSymChoice, nnkClosedSymChoice, nnkSym} and
|
2019-05-19 21:11:17 +00:00
|
|
|
$exp[0] in ["not", "in", "notin", "==", "<=",
|
|
|
|
">=", "<", ">", "!=", "is", "isnot"]:
|
|
|
|
|
|
|
|
for i in 1 ..< exp.len:
|
|
|
|
if exp[i].kind notin nnkLiterals:
|
|
|
|
inc counter
|
|
|
|
let argStr = exp[i].toStrLit
|
|
|
|
let paramAst = exp[i]
|
|
|
|
if exp[i].kind == nnkIdent:
|
|
|
|
result.printOuts.add getAst(print(argStr, paramAst))
|
2021-03-19 17:27:25 +00:00
|
|
|
if exp[i].kind in nnkCallKinds + {nnkDotExpr, nnkBracketExpr, nnkPar} and
|
|
|
|
(exp[i].typeKind notin {ntyTypeDesc} or $exp[0] notin ["is", "isnot"]):
|
2019-05-19 21:11:17 +00:00
|
|
|
let callVar = newIdentNode(":c" & $counter)
|
|
|
|
result.assigns.add getAst(asgn(callVar, paramAst))
|
|
|
|
result.check[i] = callVar
|
|
|
|
result.printOuts.add getAst(print(argStr, callVar))
|
|
|
|
if exp[i].kind == nnkExprEqExpr:
|
|
|
|
# ExprEqExpr
|
2021-03-19 17:27:25 +00:00
|
|
|
# Ident "v"
|
2019-05-19 21:11:17 +00:00
|
|
|
# IntLit 2
|
|
|
|
result.check[i] = exp[i][1]
|
2021-03-19 17:27:25 +00:00
|
|
|
if exp[i].typeKind notin {ntyTypeDesc}:
|
2019-05-19 21:11:17 +00:00
|
|
|
let arg = newIdentNode(":p" & $counter)
|
|
|
|
result.assigns.add getAst(asgn(arg, paramAst))
|
|
|
|
result.printOuts.add getAst(print(argStr, arg))
|
|
|
|
if exp[i].kind != nnkExprEqExpr:
|
|
|
|
result.check[i] = arg
|
|
|
|
else:
|
|
|
|
result.check[i][1] = arg
|
|
|
|
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
let
|
|
|
|
checkpointSym = bindSym("checkpoint")
|
|
|
|
checkSym = bindSym("check")
|
|
|
|
failSym = bindSym("fail")
|
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
case checked.kind
|
|
|
|
of nnkCallKinds:
|
|
|
|
let (assigns, check, printOuts) = inspectArgs(checked)
|
2021-03-19 17:27:25 +00:00
|
|
|
let lineinfo = newStrLitNode(checked.lineInfo)
|
2019-05-19 21:11:17 +00:00
|
|
|
let callLit = checked.toStrLit
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
let checkpointSym = bindSym("checkpoint")
|
2023-08-22 04:50:19 +00:00
|
|
|
result = nnkBlockStmt.newTree(
|
|
|
|
newEmptyNode(),
|
|
|
|
nnkStmtList.newTree(
|
|
|
|
assigns,
|
|
|
|
nnkIfStmt.newTree(
|
|
|
|
nnkElifBranch.newTree(
|
|
|
|
nnkCall.newTree(ident("not"), check),
|
|
|
|
nnkStmtList.newTree(
|
|
|
|
nnkCall.newTree(
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
checkpointSym,
|
2023-08-22 04:50:19 +00:00
|
|
|
nnkInfix.newTree(
|
|
|
|
ident("&"),
|
|
|
|
nnkInfix.newTree(
|
|
|
|
ident("&"),
|
|
|
|
lineinfo,
|
|
|
|
newLit(": Check failed: ")
|
|
|
|
),
|
|
|
|
callLit
|
|
|
|
)
|
|
|
|
),
|
|
|
|
printOuts,
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
nnkCall.newTree(failSym)
|
2023-08-22 04:50:19 +00:00
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
of nnkStmtList:
|
|
|
|
result = newNimNode(nnkStmtList)
|
|
|
|
for node in checked:
|
|
|
|
if node.kind != nnkCommentStmt:
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
result.add(newCall(checkSym, node))
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
else:
|
2021-03-19 17:27:25 +00:00
|
|
|
let lineinfo = newStrLitNode(checked.lineInfo)
|
2019-05-19 21:11:17 +00:00
|
|
|
let callLit = checked.toStrLit
|
|
|
|
|
2023-08-22 04:50:19 +00:00
|
|
|
result = nnkBlockStmt.newTree(
|
|
|
|
newEmptyNode(),
|
|
|
|
nnkStmtList.newTree(
|
|
|
|
nnkIfStmt.newTree(
|
|
|
|
nnkElifBranch.newTree(
|
|
|
|
nnkCall.newTree(ident("not"), checked),
|
|
|
|
nnkStmtList.newTree(
|
|
|
|
nnkCall.newTree(
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
checkpointSym,
|
2023-08-22 04:50:19 +00:00
|
|
|
nnkInfix.newTree(
|
|
|
|
ident("&"),
|
|
|
|
nnkInfix.newTree(
|
|
|
|
ident("&"),
|
|
|
|
lineinfo,
|
|
|
|
newLit(": Check failed: ")
|
|
|
|
),
|
|
|
|
callLit
|
|
|
|
)
|
|
|
|
),
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
nnkCall.newTree(failSym)
|
2023-08-22 04:50:19 +00:00
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
|
|
|
)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
|
|
|
template require*(conditions: untyped) =
|
|
|
|
## Same as `check` except any failed test causes the program to quit
|
|
|
|
## immediately. Any teardown statements are not executed and the failed
|
|
|
|
## test output is not generated.
|
|
|
|
let savedAbortOnError = abortOnError
|
|
|
|
block:
|
|
|
|
abortOnError = true
|
|
|
|
check conditions
|
|
|
|
abortOnError = savedAbortOnError
|
|
|
|
|
|
|
|
macro expect*(exceptions: varargs[typed], body: untyped): untyped =
|
|
|
|
## Test if `body` raises an exception found in the passed `exceptions`.
|
|
|
|
## The test passes if the raised exception is part of the acceptable
|
|
|
|
## exceptions. Otherwise, it fails.
|
2021-03-19 17:27:25 +00:00
|
|
|
runnableExamples:
|
|
|
|
import std/[math, random, strutils]
|
|
|
|
proc defectiveRobot() =
|
|
|
|
randomize()
|
|
|
|
case rand(1..4)
|
|
|
|
of 1: raise newException(OSError, "CANNOT COMPUTE!")
|
|
|
|
of 2: discard parseInt("Hello World!")
|
|
|
|
of 3: raise newException(IOError, "I can't do that Dave.")
|
|
|
|
else: assert 2 + 2 == 5
|
|
|
|
|
|
|
|
expect IOError, OSError, ValueError, AssertionDefect:
|
|
|
|
defectiveRobot()
|
|
|
|
|
2019-05-19 21:11:17 +00:00
|
|
|
template expectBody(errorTypes, lineInfoLit, body): NimNode {.dirty.} =
|
|
|
|
try:
|
2023-02-17 14:23:16 +00:00
|
|
|
try:
|
|
|
|
body
|
|
|
|
checkpoint(lineInfoLit & ": Expect Failed, no exception was thrown.")
|
|
|
|
fail()
|
|
|
|
except errorTypes:
|
|
|
|
discard
|
2023-01-04 14:48:30 +00:00
|
|
|
except CatchableError as e:
|
|
|
|
checkpoint(lineInfoLit & ": Expect Failed, unexpected " & $e.name &
|
|
|
|
" (" & e.msg & ") was thrown.\n" & e.getStackTrace())
|
|
|
|
fail()
|
|
|
|
except Defect as e:
|
2021-08-17 13:45:12 +00:00
|
|
|
checkpoint(lineInfoLit & ": Expect Failed, unexpected " & $e.name &
|
|
|
|
" (" & e.msg & ") was thrown.\n" & e.getStackTrace())
|
2019-05-19 21:11:17 +00:00
|
|
|
fail()
|
|
|
|
|
|
|
|
var errorTypes = newNimNode(nnkBracket)
|
2021-03-19 17:27:25 +00:00
|
|
|
for exp in exceptions:
|
|
|
|
errorTypes.add(exp)
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-03-19 17:27:25 +00:00
|
|
|
result = getAst(expectBody(errorTypes, errorTypes.lineInfo, body))
|
2019-05-19 21:11:17 +00:00
|
|
|
|
2021-04-28 08:19:23 +00:00
|
|
|
proc disableParamFiltering* {.deprecated:
|
2021-04-29 12:20:03 +00:00
|
|
|
"Compile with -d:unittest2DisableParamFiltering instead".} =
|
|
|
|
discard
|
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
2023-09-01 10:23:01 +00:00
|
|
|
|
|
|
|
when unittest2PreviewIsolate:
|
|
|
|
import std/[osproc, strtabs]
|
|
|
|
proc runIsolated(test: Test) =
|
|
|
|
# Run test in an isolated process - this has the advantage that we can
|
|
|
|
# trivially capture stdout but has a number of problems:
|
|
|
|
# * suite and other global stuff gets executed for each test
|
|
|
|
# * on unix, `fork` could work around this but not on windows
|
|
|
|
# * there's no good way to separate errors from stdout
|
|
|
|
# * there's process overhead
|
|
|
|
#
|
|
|
|
# There are advantages too:
|
|
|
|
# * reduced cross-test pollution
|
|
|
|
# * simple to parallelise
|
|
|
|
# * we can abort long-running tests after a timeout
|
|
|
|
|
|
|
|
let startTime = getMonoTime()
|
|
|
|
testStarted(test.testName)
|
|
|
|
|
|
|
|
let runner = startProcess(
|
|
|
|
getAppFilename2(),
|
|
|
|
args = [test.suiteName & "::" & test.testName],
|
|
|
|
env = newStringTable(
|
|
|
|
"UNITTEST2_ISOLATED", "1",
|
|
|
|
StringTableMode.modeCaseSensitive),
|
|
|
|
options = {poStdErrToStdOut})
|
|
|
|
|
|
|
|
close(runner.inputStream) # EOF so the test doesn't think it'll get input
|
|
|
|
|
|
|
|
var output: string
|
|
|
|
|
|
|
|
while true:
|
|
|
|
let pos = output.len
|
|
|
|
output.setLen(pos + 4096)
|
|
|
|
|
|
|
|
let bytes = runner.outputStream.readData(addr output[pos], 4096)
|
|
|
|
if bytes >= 0:
|
|
|
|
output.setLen(pos + bytes)
|
|
|
|
|
|
|
|
if bytes <= 0:
|
|
|
|
break
|
|
|
|
|
|
|
|
let status = runner.waitForExit()
|
|
|
|
|
|
|
|
runner.close()
|
|
|
|
|
|
|
|
testEnded(TestResult(
|
|
|
|
suiteName: test.suiteName,
|
|
|
|
testName: test.testName,
|
|
|
|
status: if status == 0: TestStatus.OK else: TestStatus.FAILED,
|
|
|
|
duration: getMonoTime() - startTime,
|
|
|
|
output: output
|
|
|
|
))
|
|
|
|
|
|
|
|
type
|
|
|
|
IsolatedFormatter* = ref object of OutputFormatter
|
|
|
|
## Formatter suitable for using the process-isolated environment
|
|
|
|
##
|
|
|
|
## This is a work in progress with several open issues
|
|
|
|
## * we could use stderr for "unittest" traffic but it would be
|
|
|
|
## compromised by application output (typically ok in nim) and makes
|
|
|
|
## reading messy
|
|
|
|
## * we could print all errors after test providing some sort of
|
|
|
|
## separator - has escape issues
|
|
|
|
## * we could redirect stdout/stderr to a file and use stdout for errors
|
|
|
|
## * as an addon to the above, we could read back the file then print
|
|
|
|
## a structured test format to stdout which the parent process can
|
|
|
|
## capture easily
|
|
|
|
|
|
|
|
if isolated:
|
|
|
|
formatters.add(IsolatedFormatter())
|
|
|
|
|
|
|
|
method failureOccurred*(formatter: IsolatedFormatter,
|
|
|
|
checkpoints: seq[string], stackTrace: string) =
|
|
|
|
if stackTrace.len > 0:
|
|
|
|
echo(stackTrace)
|
|
|
|
echo("\n")
|
|
|
|
for msg in items(checkpoints):
|
|
|
|
echo(" ")
|
|
|
|
echo(msg)
|
|
|
|
echo("\n")
|
|
|
|
|
|
|
|
when collect:
|
|
|
|
proc runScheduledTests() {.noconv.} =
|
|
|
|
# Tests can be added inside tests - this is weird and only partially
|
|
|
|
# supported
|
|
|
|
while tests.len > 0:
|
|
|
|
var tmp = move(tests)
|
|
|
|
suiteRunStarted(tmp)
|
|
|
|
for suiteName, suite in tmp:
|
|
|
|
if suite.len == 0: continue
|
|
|
|
|
|
|
|
suiteStarted(suiteName)
|
|
|
|
for test in suite:
|
|
|
|
when isolate:
|
|
|
|
if not isolated:
|
|
|
|
runIsolated(test)
|
|
|
|
else:
|
|
|
|
runDirect(test)
|
|
|
|
else:
|
|
|
|
runDirect(test)
|
|
|
|
|
|
|
|
suiteEnded()
|
|
|
|
|
|
|
|
suiteRunEnded()
|
|
|
|
testRunEnded()
|
|
|
|
|
|
|
|
addExitProc(runScheduledTests)
|
|
|
|
|
|
|
|
else:
|
|
|
|
addExitProc(proc() {.noconv.} = testRunEnded())
|