The main issue was caused by an underflow in the partial payouts calculation. The underflow has been fixed.
The rest is creation of the test and supporting changes for the test.
split codex_testing compilation flag into two flags:
1. codex_enable_proof_failures - used for simulating proof failures in the tests
2. codex_use_hardhat - used for a workaround to a Hardhat bug that obtains the current block for clock.now in the OnChainClock
solved an issue on windows where the processes were not exiting cleanly and causing an attempted write to the dht db after the data dir had already been deleted
- removed ethersuite as the base testing suite for multinodesuite because the need to launch hardhat before ethersuite setup and kill hardhat after ethersuite teardown was required. Removing ethersuite allowed for checking if a connection to hardhat was established. If no connection established, hardhat would need to be started via the test config. If there was a connection established, a snapshot is taken during test setup and reverted during teardown.
- modified the way the hardhat process was launched, because evaluating a command using chronos startProcess would wait for the command to complete before access to the output stream was given. Instead, we now launch the hardhat executable from node_modules/.bin/hardhat.
- modify the way the processes are exited by killing them immediately.
- fix warnings
- Use rest api check to determine when node started
- Use poInteractive and poStdErrToStdOut process start options in the hopes it may help with the small windows buffers
Due to a bug in hardhat, the latest block timestamp was out of sync from the newHeads block timestamp which was causing some timing issues in the tests. To fix this, a compile flag, `codex_testing` (which was changed from `codex_enable_proof_failures`) is used to determine the bevhaviour of clock.now. If `codex_testing` is true, clock.now uses the latest block timestamp, else it uses the cached block.timestamp that was populated in the `newHeads` event.
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
Eric