This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
* add counters to xml reports
* add command line args to write xml report
* fix initialization being performed once for each test
* use monotonic timer for duration measurements
Diego
jangko
Jacek Sieka
tersec
Tanguy
narimiran
Ivan Yonchovski
Ștefan Talpalaru