- Move test manager values to config object
- Increase codex port separation (between api and disc ports) in an attempt to prevent overlap across tests (ie Test1: api=8000(tcp), disc=9000(udp), and Test2: api=9000(tcp), disc=10000(udp))
- print stderr when exitcode == 1 if there's < 3 lines of stdout
- Logging:
- Always write test manager harness chronicles logs to file, ie testmanager.chronicles.log in the root of the `integration/logs/<run name>` dir
- Always write individual test stdout to file, ie `<test file name>.stdout.log` in the root of the `integration/logs/<run name>/<test file name>` dir
- On error, print stderr to screen and write stderr to file. Or on failure, if stdout is sufficiently short, write stderr to screen and file in `integration/logs/<run name>/<test file name>/<test file name>.stderr.log`
- When debugging, ie DebugCodexNodes == true
- Removes DebugTestHarness from controlling anything other than printing chronicles output from the testmanager to the terminal
- Now, if DebugCodexNodes is set to true:
- Codex node (chronicles) output for multinodesuite tests is logged to file, eg `integration/logs/<run name>/<test file name>/<test name>/<role>_<idx>.log`
- Codex chronicles output is logged to stdout, which also written to file (see above)
- test harness chronicles logs to log file
- test process stdout and stderr to log file for each integration test file
Note: does not work when DEBUG=1 is not present in make command
- set disabled topics during codex compilation due to lack of support for disabling chronicles topics at runtime in codex
- ensure that TRACE logs are set for nodes running in multinodesuite when DebugCodexNodes is set
- add TRACE loglevel to testcli
Codex api/disc ports were clashing in windows because `lastUsedCodexApiPort` and `lastUsedCodeexDiscPort` were not locked when read and written, while client and provider nodes were started concurrently (async).
DEBUG -> enables DebugCodexNodes, DebugTestHarness, and ShowContinuousStatusUpdates
DEBUG_HARDHAT -> enables DebugHardhat
PARALLEL -> enables EnableParallelTests
Additionally, when DEBUG is enabled, all integration tests debug configs are enabled for Codex nodes, the Codex node output is printed with the test output (not interleaved), and the Codex node output is logged to file in `tests/integrations/logs/<starttime>__IntegrationTests/<integration_test_name>/<suite_name>/<testname>/<role>_<idx>.log`.
When DEBUG_HARDHAT is enabled, all hardhat output is printed with the test output (not interleaved), and the output is also written to a log file in `tests/integrations/logs/<starttime>__IntegrationTests/<integration_test_name>/hardhat.log
Previously, each test would start with a port specified by a global const and overwritable using `define` cli flag. This seems to be causing some port clashes that `nextFreePort` is not picking up. Instead of relying on `nextFreePort`, do not reuse port numbers for any of the tests.
Solution was two-fold:
1. Run the hardhat.cmd script instead of hardhat
2. Use the absolute path to the script executable instead of relative to the working directory
- bubble errors from hardhatprocess.start (instead of just logging)
- push raises: [] in all nodeprocess procs/methods to avoid leaking `Exception` exception types in method overrides
