In situations like timeouts, windows will hang when attempting to close the test process streams. In this case, we have to force kill the test process externally. This is the same process as force killing hardhat nodes after they are already terminated, but windows refuses hangs when closing their process streams. This commit creates a forceKillProcess utility that allows a process to be killed by its process name and matching commandline criteria, like TestId (for test process) or --port (for hardhat)
Hardhat output is logged to file in hardhat.log for each test, and printing to screen is not necessarily needed as it is already logged to file and can create clutter in the output, so stop writing logging output to memory and writing to screen.
- Codex nodes will log to file by default, so this parameter could be removed.
- Add NoCodexLogFilters define, that unsets chronicles' topic filters when logging codex nodes chronicles logs to file
- Move test manager values to config object
- Increase codex port separation (between api and disc ports) in an attempt to prevent overlap across tests (ie Test1: api=8000(tcp), disc=9000(udp), and Test2: api=9000(tcp), disc=10000(udp))
- print stderr when exitcode == 1 if there's < 3 lines of stdout
- Logging:
- Always write test manager harness chronicles logs to file, ie testmanager.chronicles.log in the root of the `integration/logs/<run name>` dir
- Always write individual test stdout to file, ie `<test file name>.stdout.log` in the root of the `integration/logs/<run name>/<test file name>` dir
- On error, print stderr to screen and write stderr to file. Or on failure, if stdout is sufficiently short, write stderr to screen and file in `integration/logs/<run name>/<test file name>/<test file name>.stderr.log`
- When debugging, ie DebugCodexNodes == true
- Removes DebugTestHarness from controlling anything other than printing chronicles output from the testmanager to the terminal
- Now, if DebugCodexNodes is set to true:
- Codex node (chronicles) output for multinodesuite tests is logged to file, eg `integration/logs/<run name>/<test file name>/<test name>/<role>_<idx>.log`
- Codex chronicles output is logged to stdout, which also written to file (see above)
- test harness chronicles logs to log file
- test process stdout and stderr to log file for each integration test file
Note: does not work when DEBUG=1 is not present in make command
DEBUG -> enables DebugCodexNodes, DebugTestHarness, and ShowContinuousStatusUpdates
DEBUG_HARDHAT -> enables DebugHardhat
PARALLEL -> enables EnableParallelTests
Additionally, when DEBUG is enabled, all integration tests debug configs are enabled for Codex nodes, the Codex node output is printed with the test output (not interleaved), and the Codex node output is logged to file in `tests/integrations/logs/<starttime>__IntegrationTests/<integration_test_name>/<suite_name>/<testname>/<role>_<idx>.log`.
When DEBUG_HARDHAT is enabled, all hardhat output is printed with the test output (not interleaved), and the output is also written to a log file in `tests/integrations/logs/<starttime>__IntegrationTests/<integration_test_name>/hardhat.log
- add a TestManager property to IntegrationTest, so manager does not need to be passed into all functions
- cleanup:
- remove unneeded stopHardhat function
- add hardhat instance to manager outside of startHardhat
