Commit Graph

5 Commits

Author SHA1 Message Date
Jacek Sieka 2300fa9924
Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!

By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:

```
[  20/25] HTTP client testing suite                               ................s..... (14.2s)
[  21/25] Asynchronous process management test suite              ....................F. (14.5s)
===========================
  /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
    /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
    getCurrentFD() was 3
    markFD was 5

  [FAILED ] (  0.00s) File descriptors leaks test
```

For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.

Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.

The verbose mode has also received an upgrade:

```
  [  20/22]           terminateAndWaitForExit() timeout test
  [OK     ] (  2.00s) terminateAndWaitForExit() timeout test
  [  21/22]           File descriptors leaks test
===========================
  /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
    /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
    getCurrentFD() was 3
    markFD was 5

  [FAILED ] (  0.00s) File descriptors leaks test
  [  22/22]           Leaks test
```

Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.

At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:

```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
  [FAILED ] (  0.00s) File descriptors leaks test
```

As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!

The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.

Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.

The changes also come with a bag of mixed stuff:

* the parallel mode is removed entirely - it was broken beyond fixing and
  needs to be rewritten from zero - deadlocks, GC violations and everything
  in between rendered it practically unusable and a source of CI failures
  above all
* an experimental process isolation mode where tests are run in a separate
  process - this is allows many features such as timeouts, output
  verification etc but it also breaks for similar reasons as above:
  "global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
  line options
2023-09-01 12:23:01 +02:00
Ștefan Talpalaru 24eb5e829e
CI: test with Nim-1.6 and C++ (#11) 2022-01-14 04:15:45 +01:00
Ștefan Talpalaru 66420a7d93
regenerate HTML docs 2021-11-22 01:08:32 +01:00
Ștefan Talpalaru b9b61cfe4a
change define prefix and update docs (#7) 2021-04-29 14:20:03 +02:00
Ștefan Talpalaru 50c3aff5be
CI setup
Tests moved to "config.nims", to avoid building and using Nimble.

fix for >=Nim-1.4.0

Nim devel fix

macOS fix

fix Windows (uninitialised locks)
2021-03-19 23:22:19 +01:00