Revamped output handling, cleanup and removal of thread mode (#31)
This PR introduces a facelift for `unittest2` bringing its UX into the 21st
century!
By default, a compact dot-based format is used to avoid spamming the console
on success - successes are normal and only minimal information is needed in
the vast majority of runs:
```
[ 20/25] HTTP client testing suite ................s..... (14.2s)
[ 21/25] Asynchronous process management test suite ....................F. (14.5s)
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
```
For each line, we see a suite followed by single-character status markers -
mostly successes but also a skipped and a failed test.
Failures are printed verbosely after the suite is done so they don't get
lost in the success spam along with a simple copy-pasteable line showing
how to reproduce the failure.
The verbose mode has also received an upgrade:
```
[ 20/22] terminateAndWaitForExit() timeout test
[OK ] ( 2.00s) terminateAndWaitForExit() timeout test
[ 21/22] File descriptors leaks test
===========================
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test'
---------------------------
/home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD
getCurrentFD() was 3
markFD was 5
[FAILED ] ( 0.00s) File descriptors leaks test
[ 22/22] Leaks test
```
Here, we can see a "test started" marker so that it becomes clear which test
is currently running and so that any output the test itself prints has the
proper name attached to it.
At the end, there's a summary as well that reiterates the tests that failed so
they can be found in the success spam that the verbose mode generates:
```
[Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED
[FAILED ] ( 0.00s) File descriptors leaks test
```
As seen above, the new mode knows how many tests will run: this is thanks
to a new two-phase mode of test running: "collect" and "run"!
The "collection" phase is responsible for collecting test metadata which is
later used to run tests in smarter ways, for example in isolated processes.
Unfortunately, it is not fully backwards-compatible because the global setup
might expose order-dependency problems in test suites that previously would
not be visible - for this, we have acompile-time setting for a compatibilty
mode that runs things roughly in the same order as before but disables
some of the fancy functionality.
The changes also come with a bag of mixed stuff:
* the parallel mode is removed entirely - it was broken beyond fixing and
needs to be rewritten from zero - deadlocks, GC violations and everything
in between rendered it practically unusable and a source of CI failures
above all
* an experimental process isolation mode where tests are run in a separate
process - this is allows many features such as timeouts, output
verification etc but it also breaks for similar reasons as above:
"global" code gets executed out of order and repeatedly.
* random UX fixes and cleanups of things like env var reading and command
line options
This commit is contained in:
Jacek Sieka
GitHub
parent
299bc9a574
commit
2300fa9924
46
README.md
46
README.md
|
|
@ -4,21 +4,23 @@
|
|||
|
||||
Features of `unittest2` include:
|
||||
|
||||
* [Parallel test execution](https://status-im.github.io/nim-unittest2/unittest2.html#running-tests-in-parallel)
|
||||
* Beautiful and compact user experience adapted for both humans and CI
|
||||
* Test separation with each test running in its own procedure
|
||||
* Strict exception handling with support for [exception tracking](https://nim-lang.org/docs/manual.html#effect-system-exception-tracking)
|
||||
* JUnit-compatible XML test reports for tooling integration
|
||||
* Two-phase execution model as building block for advanced test scheduling and reporting features
|
||||
|
||||
`unittest2` started as a [pull request](https://github.com/nim-lang/Nim/pull/9724) to evolve the [unittest](https://nim-lang.org/docs/unittest.html) module in Nim and has since grown into a separate library.
|
||||
`unittest2` is an evolution of the [unittest](https://nim-lang.org/docs/unittest.html) module in Nim - porting, while not trivial should at least be easy.
|
||||
|
||||
## Installing
|
||||
|
||||
```text
|
||||
```sh
|
||||
nimble install unittest2
|
||||
```
|
||||
|
||||
or add a dependency in your `.nimble` file:
|
||||
|
||||
```text
|
||||
```nim
|
||||
requires "unittest2"
|
||||
```
|
||||
|
||||
|
|
@ -37,16 +39,46 @@ suite "Suites can be used to group tests":
|
|||
```
|
||||
|
||||
Compile and run the unit tests:
|
||||
```bash
|
||||
|
||||
```sh
|
||||
nim c -r test.nim
|
||||
```
|
||||
|
||||
The generated tests have a few command line options that can be viewed with `--help`:
|
||||
|
||||
```sh
|
||||
nim c -r test.nim --help
|
||||
```
|
||||
|
||||
See the [tests](./tests) for more examples!
|
||||
|
||||
## Operation
|
||||
|
||||
`unittest2` has two modes of operation: two-phase "collect-and-run" and single-pass "compatiblity".
|
||||
|
||||
The single-pass mode is currently default and broadly similar to how `unittest` works: suites and tests are run in the order they are encountered in the test files.
|
||||
|
||||
In "collect" mode a two-phase runner is used, first making a discovery pass to collect all tests then running them in a separate execution phase - this allows features such as test listing, progress indicators and in the future, smarter test scheduling strategies involving parallel and fully isolated execution.
|
||||
|
||||
The two-phase "collect" mode runs into a few notable incompatibilites with respect to the traditional `unittest` model:
|
||||
|
||||
* globals and code inside `suite` but not part of `setup`, `test` etc runs for all modules before tests are run
|
||||
* this change in execution order may result in test failures and odd performance quirks
|
||||
* when re-running tests with filters, the globals end up being processed before filtering - this problem affects `unittest` also
|
||||
* running with process isolation may lead to surprises such as resource conflicts (sockets, databases, files ..) and poor performance as both the "collection" process and the execution process ends up running the same global section of the code
|
||||
|
||||
Porting code to the two-phase mode includes:
|
||||
|
||||
* moving code in `suite` into `setup`, `teardown` and similar locations
|
||||
* removing order-dependency from tests ensuring that each test can be run independently
|
||||
|
||||
The two-phase mode will at some point become the default execution model for `unittest2` - it is enabled when the compatibility mode is turned off by passing `-d:unittest2Compat=false` to the compilation process.
|
||||
|
||||
## Porting code from `unittest`
|
||||
|
||||
* Replace `import unittest` with `import unittest2`
|
||||
* `unittest2` places each test in a separate `proc` which changes the way templates inside tests are interpreted - some code changes may be necessary
|
||||
* prepare the code for two-phase operation by reducing reliance on globals and test execution order
|
||||
|
||||
## Testing `unittest2`
|
||||
|
||||
|
|
@ -63,6 +95,8 @@ MIT
|
|||
|
||||
- original author: Zahary Karadjov
|
||||
|
||||
- fork author: Ștefan Talpalaru \<stefantalpalaru@yahoo.com\>
|
||||
- initial fork author: Ștefan Talpalaru \<stefantalpalaru@yahoo.com\>
|
||||
|
||||
- current maintainer: Status R&D (https://status.im).
|
||||
|
||||
- homepage: https://github.com/status-im/nim-unittest2
|
||||
|
|
|
|||
59
config.nims
59
config.nims
|
|
@ -1,24 +1,49 @@
|
|||
task test, "Run tests":
|
||||
let nimc = getEnv("NIMC", "nim") # Which nim compiler to use
|
||||
let lang = getEnv("NIMLANG", "c") # Which backend (c/cpp/js)
|
||||
let flags = getEnv("NIMFLAGS", "") # Extra flags for the compiler
|
||||
let verbose = getEnv("V", "") notin ["", "0"]
|
||||
|
||||
from os import quoteShell
|
||||
|
||||
let cfg =
|
||||
" --styleCheck:usages --styleCheck:error" &
|
||||
(if verbose: "" else: " --verbosity:0 --hints:off") &
|
||||
" --skipParentCfg --skipUserCfg --outdir:build -f " &
|
||||
quoteShell("--nimcache:build/nimcache/$projectName")
|
||||
|
||||
proc build(args, path: string, cmdArgs = "") =
|
||||
exec nimc & " " & lang & " " & cfg & " " & flags & " " & args & " " & path & " " & cmdArgs
|
||||
|
||||
proc run(args, path: string, cmdArgs = "") =
|
||||
build args & " -r", path, cmdArgs
|
||||
|
||||
proc testOptions() =
|
||||
let
|
||||
xmlFile = "test_results.xml"
|
||||
commandStart = "nim " & getEnv("TEST_LANG", "c") & " -r -f --threads:on --hints:off --verbosity:0 --skipParentCfg:on --skipUserCfg:on " & getEnv("NIMFLAGS") & " "
|
||||
xmlFile = "build/test_results.xml"
|
||||
rmFile xmlFile
|
||||
|
||||
# This should generate an XML results file.
|
||||
run("", "tests/tunittest", "--xml:" & xmlFile)
|
||||
doAssert fileExists xmlFile
|
||||
rmFile xmlFile
|
||||
|
||||
# This should not, since we disable param processing.
|
||||
run("-d:unittest2DisableParamFiltering", "tests/tunittest", "--xml:" & xmlFile)
|
||||
doAssert not fileExists xmlFile
|
||||
|
||||
task test, "Run tests":
|
||||
if not dirExists "build":
|
||||
mkDir "build"
|
||||
|
||||
for f in listFiles("tests"):
|
||||
if f.len > 4 and f[^4..^1] == ".nim":
|
||||
# This should generate an XML results file.
|
||||
var cmd = commandStart & f & " --xml:" & xmlFile & " --console"
|
||||
echo cmd
|
||||
exec cmd
|
||||
doAssert fileExists xmlFile
|
||||
rmFile xmlFile
|
||||
if not (f.len > 4 and f[^4..^1] == ".nim"): continue
|
||||
|
||||
# This should not, since we disable param processing.
|
||||
cmd = commandStart & "-d:unittest2DisableParamFiltering " & f & " --xml:" & xmlFile
|
||||
echo cmd
|
||||
exec cmd
|
||||
doAssert not fileExists xmlFile
|
||||
rmFile f[0..^5].toExe
|
||||
for compat in ["-d:unittest2Compat=false", "-d:unittest2Compat=true"]:
|
||||
for color in ["-d:nimUnittestColor=on", "-d:nimUnittestColor=off"]:
|
||||
for level in ["VERBOSE", "COMPACT", "FAILURES", "NONE"]:
|
||||
run "--threads:on " & " " & compat & " " & color, f, "--output-level=" & level
|
||||
|
||||
testOptions()
|
||||
|
||||
task buildDocs, "Build docs":
|
||||
exec "nim doc --skipParentCfg:on --skipUserCfg:on --outdir:docs --git.url:https://github.com/status-im/nim-unittest2 --git.commit:master --git.devel:master unittest2.nim"
|
||||
|
||||
|
|
|
|||
8
nim.cfg
8
nim.cfg
|
|
@ -1,10 +1,4 @@
|
|||
@if release:
|
||||
nimcache = "nimcache/release/$projectName"
|
||||
@else:
|
||||
nimcache = "nimcache/debug/$projectName"
|
||||
@end
|
||||
|
||||
--threads:on
|
||||
nimcache = "build/nimcache/$projectName"
|
||||
|
||||
# Avoid some rare stack corruption while using exceptions with a SEH-enabled
|
||||
# toolchain: https://github.com/status-im/nimbus-eth2/issues/3121
|
||||
|
|
|
|||
|
|
@ -1,134 +0,0 @@
|
|||
discard """
|
||||
output: '''[Suite] suite #1
|
||||
|
||||
[Suite] suite #2
|
||||
'''
|
||||
"""
|
||||
|
||||
# Unfortunately, it's not possible to decouple the thread execution order from
|
||||
# the number of available cores, due to how threadpool dynamically (and lazily)
|
||||
# adjusts the number of worker threads, so we can't have a PRINT_ALL output in
|
||||
# the verification section above.
|
||||
|
||||
import unittest2, os
|
||||
|
||||
test "independent test #1":
|
||||
sleep(1000)
|
||||
check 1 == 1
|
||||
# check 1 == 2
|
||||
# require 1 == 2
|
||||
# var g {.global.}: seq[int]
|
||||
# g.add(1)
|
||||
|
||||
test "independent test #2":
|
||||
sleep(800)
|
||||
check 1 == 1
|
||||
|
||||
test "independent test #3":
|
||||
## nested tests
|
||||
# we might as well keep this futile attempt at finding a problem with
|
||||
# uninitialized `flowVars` in child threads
|
||||
test "independent test #4":
|
||||
test "independent test #5":
|
||||
test "independent test #8":
|
||||
test "independent test #9":
|
||||
test "independent test #10":
|
||||
test "independent test #11":
|
||||
test "independent test #12":
|
||||
test "independent test #13":
|
||||
test "independent test #14":
|
||||
test "independent test #15":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #16":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #17":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #18":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #19":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #20":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #21":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #22":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #23":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #24":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
test "independent test #25":
|
||||
test "independent test #26":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
sleep(400)
|
||||
check 1 == 1
|
||||
sleep(600)
|
||||
check 1 == 1
|
||||
|
||||
suite "suite #1":
|
||||
test "suite #1, test #1":
|
||||
sleep(400)
|
||||
check 1 == 1
|
||||
|
||||
test "suite #1, test #2":
|
||||
sleep(300)
|
||||
check 1 == 1
|
||||
|
||||
# change the output formatter just for the next suite
|
||||
resetOutputFormatters()
|
||||
addOutputFormatter(newConsoleOutputFormatter(colorOutput=false))
|
||||
|
||||
suite "suite #2":
|
||||
test "suite #2, test #1":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
|
||||
test "suite #2, test #2":
|
||||
sleep(100)
|
||||
check 1 == 1
|
||||
|
||||
suiteTeardown:
|
||||
echo "suite teardown"
|
||||
|
||||
echo "this will be shown first"
|
||||
|
||||
# go back to the default one
|
||||
resetOutputFormatters()
|
||||
|
||||
test "independent test #6":
|
||||
sleep(200)
|
||||
check 1 == 1
|
||||
|
||||
test "independent test #7":
|
||||
sleep(100)
|
||||
check 1 == 1
|
||||
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
-d:nimtestParallel
|
||||
|
||||
1007
unittest2.nim
1007
unittest2.nim
File diff suppressed because it is too large
Load Diff
|
|
@ -1,8 +1,7 @@
|
|||
mode = ScriptMode.Verbose
|
||||
|
||||
version = "0.0.9"
|
||||
version = "0.1.0"
|
||||
author = "Status Research & Development GmbH"
|
||||
description = "unittest fork with support for parallel test execution"
|
||||
license = "MIT"
|
||||
requires "nim >= 1.6.0"
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue