nim-unittest2/config.nims

50 lines
1.8 KiB
Plaintext
Raw Normal View History

Revamped output handling, cleanup and removal of thread mode (#31) This PR introduces a facelift for `unittest2` bringing its UX into the 21st century! By default, a compact dot-based format is used to avoid spamming the console on success - successes are normal and only minimal information is needed in the vast majority of runs: ``` [ 20/25] HTTP client testing suite ................s..... (14.2s) [ 21/25] Asynchronous process management test suite ....................F. (14.5s) =========================== /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test' --------------------------- /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD getCurrentFD() was 3 markFD was 5 [FAILED ] ( 0.00s) File descriptors leaks test ``` For each line, we see a suite followed by single-character status markers - mostly successes but also a skipped and a failed test. Failures are printed verbosely after the suite is done so they don't get lost in the success spam along with a simple copy-pasteable line showing how to reproduce the failure. The verbose mode has also received an upgrade: ``` [ 20/22] terminateAndWaitForExit() timeout test [OK ] ( 2.00s) terminateAndWaitForExit() timeout test [ 21/22] File descriptors leaks test =========================== /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test' --------------------------- /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD getCurrentFD() was 3 markFD was 5 [FAILED ] ( 0.00s) File descriptors leaks test [ 22/22] Leaks test ``` Here, we can see a "test started" marker so that it becomes clear which test is currently running and so that any output the test itself prints has the proper name attached to it. At the end, there's a summary as well that reiterates the tests that failed so they can be found in the success spam that the verbose mode generates: ``` [Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED [FAILED ] ( 0.00s) File descriptors leaks test ``` As seen above, the new mode knows how many tests will run: this is thanks to a new two-phase mode of test running: "collect" and "run"! The "collection" phase is responsible for collecting test metadata which is later used to run tests in smarter ways, for example in isolated processes. Unfortunately, it is not fully backwards-compatible because the global setup might expose order-dependency problems in test suites that previously would not be visible - for this, we have acompile-time setting for a compatibilty mode that runs things roughly in the same order as before but disables some of the fancy functionality. The changes also come with a bag of mixed stuff: * the parallel mode is removed entirely - it was broken beyond fixing and needs to be rewritten from zero - deadlocks, GC violations and everything in between rendered it practically unusable and a source of CI failures above all * an experimental process isolation mode where tests are run in a separate process - this is allows many features such as timeouts, output verification etc but it also breaks for similar reasons as above: "global" code gets executed out of order and repeatedly. * random UX fixes and cleanups of things like env var reading and command line options
2023-09-01 10:23:01 +00:00
let nimc = getEnv("NIMC", "nim") # Which nim compiler to use
let lang = getEnv("NIMLANG", "c") # Which backend (c/cpp/js)
let flags = getEnv("NIMFLAGS", "") # Extra flags for the compiler
let verbose = getEnv("V", "") notin ["", "0"]
from os import quoteShell
let cfg =
" --styleCheck:usages --styleCheck:error" &
(if verbose: "" else: " --verbosity:0 --hints:off") &
" --skipParentCfg --skipUserCfg --outdir:build -f " &
quoteShell("--nimcache:build/nimcache/$projectName")
proc build(args, path: string, cmdArgs = "") =
exec nimc & " " & lang & " " & cfg & " " & flags & " " & args & " " & path & " " & cmdArgs
proc run(args, path: string, cmdArgs = "") =
build args & " -r", path, cmdArgs
proc testOptions() =
let
Revamped output handling, cleanup and removal of thread mode (#31) This PR introduces a facelift for `unittest2` bringing its UX into the 21st century! By default, a compact dot-based format is used to avoid spamming the console on success - successes are normal and only minimal information is needed in the vast majority of runs: ``` [ 20/25] HTTP client testing suite ................s..... (14.2s) [ 21/25] Asynchronous process management test suite ....................F. (14.5s) =========================== /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test' --------------------------- /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD getCurrentFD() was 3 markFD was 5 [FAILED ] ( 0.00s) File descriptors leaks test ``` For each line, we see a suite followed by single-character status markers - mostly successes but also a skipped and a failed test. Failures are printed verbosely after the suite is done so they don't get lost in the success spam along with a simple copy-pasteable line showing how to reproduce the failure. The verbose mode has also received an upgrade: ``` [ 20/22] terminateAndWaitForExit() timeout test [OK ] ( 2.00s) terminateAndWaitForExit() timeout test [ 21/22] File descriptors leaks test =========================== /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test' --------------------------- /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD getCurrentFD() was 3 markFD was 5 [FAILED ] ( 0.00s) File descriptors leaks test [ 22/22] Leaks test ``` Here, we can see a "test started" marker so that it becomes clear which test is currently running and so that any output the test itself prints has the proper name attached to it. At the end, there's a summary as well that reiterates the tests that failed so they can be found in the success spam that the verbose mode generates: ``` [Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED [FAILED ] ( 0.00s) File descriptors leaks test ``` As seen above, the new mode knows how many tests will run: this is thanks to a new two-phase mode of test running: "collect" and "run"! The "collection" phase is responsible for collecting test metadata which is later used to run tests in smarter ways, for example in isolated processes. Unfortunately, it is not fully backwards-compatible because the global setup might expose order-dependency problems in test suites that previously would not be visible - for this, we have acompile-time setting for a compatibilty mode that runs things roughly in the same order as before but disables some of the fancy functionality. The changes also come with a bag of mixed stuff: * the parallel mode is removed entirely - it was broken beyond fixing and needs to be rewritten from zero - deadlocks, GC violations and everything in between rendered it practically unusable and a source of CI failures above all * an experimental process isolation mode where tests are run in a separate process - this is allows many features such as timeouts, output verification etc but it also breaks for similar reasons as above: "global" code gets executed out of order and repeatedly. * random UX fixes and cleanups of things like env var reading and command line options
2023-09-01 10:23:01 +00:00
xmlFile = "build/test_results.xml"
rmFile xmlFile
# This should generate an XML results file.
run("", "tests/tunittest", "--xml:" & xmlFile)
doAssert fileExists xmlFile
rmFile xmlFile
# This should not, since we disable param processing.
run("-d:unittest2DisableParamFiltering", "tests/tunittest", "--xml:" & xmlFile)
doAssert not fileExists xmlFile
task test, "Run tests":
if not dirExists "build":
mkDir "build"
for f in listFiles("tests"):
Revamped output handling, cleanup and removal of thread mode (#31) This PR introduces a facelift for `unittest2` bringing its UX into the 21st century! By default, a compact dot-based format is used to avoid spamming the console on success - successes are normal and only minimal information is needed in the vast majority of runs: ``` [ 20/25] HTTP client testing suite ................s..... (14.2s) [ 21/25] Asynchronous process management test suite ....................F. (14.5s) =========================== /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/build/testall 'Asynchronous process management test suite::File descriptors leaks test' --------------------------- /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD getCurrentFD() was 3 markFD was 5 [FAILED ] ( 0.00s) File descriptors leaks test ``` For each line, we see a suite followed by single-character status markers - mostly successes but also a skipped and a failed test. Failures are printed verbosely after the suite is done so they don't get lost in the success spam along with a simple copy-pasteable line showing how to reproduce the failure. The verbose mode has also received an upgrade: ``` [ 20/22] terminateAndWaitForExit() timeout test [OK ] ( 2.00s) terminateAndWaitForExit() timeout test [ 21/22] File descriptors leaks test =========================== /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testall 'Asynchronous process management test suite::File descriptors leaks test' --------------------------- /home/arnetheduck/status/nimbus-eth2/vendor/nim-chronos/tests/testproc.nim(465, 27): Check failed: getCurrentFD() == markFD getCurrentFD() was 3 markFD was 5 [FAILED ] ( 0.00s) File descriptors leaks test [ 22/22] Leaks test ``` Here, we can see a "test started" marker so that it becomes clear which test is currently running and so that any output the test itself prints has the proper name attached to it. At the end, there's a summary as well that reiterates the tests that failed so they can be found in the success spam that the verbose mode generates: ``` [Summary ] 22 tests run: 21 OK, 1 FAILED, 0 SKIPPED [FAILED ] ( 0.00s) File descriptors leaks test ``` As seen above, the new mode knows how many tests will run: this is thanks to a new two-phase mode of test running: "collect" and "run"! The "collection" phase is responsible for collecting test metadata which is later used to run tests in smarter ways, for example in isolated processes. Unfortunately, it is not fully backwards-compatible because the global setup might expose order-dependency problems in test suites that previously would not be visible - for this, we have acompile-time setting for a compatibilty mode that runs things roughly in the same order as before but disables some of the fancy functionality. The changes also come with a bag of mixed stuff: * the parallel mode is removed entirely - it was broken beyond fixing and needs to be rewritten from zero - deadlocks, GC violations and everything in between rendered it practically unusable and a source of CI failures above all * an experimental process isolation mode where tests are run in a separate process - this is allows many features such as timeouts, output verification etc but it also breaks for similar reasons as above: "global" code gets executed out of order and repeatedly. * random UX fixes and cleanups of things like env var reading and command line options
2023-09-01 10:23:01 +00:00
if not (f.len > 4 and f[^4..^1] == ".nim"): continue
for compat in ["-d:unittest2Compat=false", "-d:unittest2Compat=true"]:
for color in ["-d:nimUnittestColor=on", "-d:nimUnittestColor=off"]:
for level in ["VERBOSE", "COMPACT", "FAILURES", "NONE"]:
run "--threads:on " & " " & compat & " " & color, f, "--output-level=" & level
testOptions()
task buildDocs, "Build docs":
2021-11-22 00:08:32 +00:00
exec "nim doc --skipParentCfg:on --skipUserCfg:on --outdir:docs --git.url:https://github.com/status-im/nim-unittest2 --git.commit:master --git.devel:master unittest2.nim"