nimbus-eth1/tests/test_clique.nim

365 lines
12 KiB
Nim
Raw Normal View History

# Nimbus
# Copyright (c) 2018-2019 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed except
# according to those terms.
import
std/[algorithm, os, sequtils, strformat, strutils, times],
chronicles,
2022-12-02 04:39:12 +00:00
eth/keys,
stint,
unittest2,
2022-12-02 04:39:12 +00:00
../nimbus/core/[chain,
clique,
clique/clique_snapshot,
clique/clique_desc,
clique/clique_helpers
],
2022-12-02 04:39:12 +00:00
../nimbus/common/[common,context],
../nimbus/utils/[ec_recover, utils],
../nimbus/[config, constants],
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
./test_clique/pool,
./replay/undump_blocks
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
const
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
baseDir = [".", "tests", ".." / "tests", $DirSep] # path containg repo
repoDir = ["test_clique", "replay", "status"] # alternative repos
goerliCapture = "goerli68161.txt.gz"
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
groupReplayTransactions = 7
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
# ------------------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------------------
proc getBlockHeader(ap: TesterPool; number: BlockNumber): BlockHeader =
## Shortcut => db/db_chain.getBlockHeader()
doAssert ap.db.getBlockHeader(number, result)
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
proc ppSecs(elapsed: Duration): string =
result = $elapsed.inSeconds
let ns = elapsed.inNanoseconds mod 1_000_000_000
if ns != 0:
# to rounded decimal seconds
let ds = (ns + 5_000_000i64) div 10_000_000i64
result &= &".{ds:02}"
result &= "s"
proc ppRow(elapsed: Duration): string =
let ms = elapsed.inMilliSeconds + 500
"x".repeat(ms div 1000)
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
proc findFilePath(file: string): string =
result = "?unknown?" / file
for dir in baseDir:
for repo in repoDir:
let path = dir / repo / file
if path.fileExists:
return path
proc setTraceLevel =
discard
when defined(chronicles_runtime_filtering) and loggingEnabled:
setLogLevel(LogLevel.TRACE)
proc setErrorLevel =
discard
when defined(chronicles_runtime_filtering) and loggingEnabled:
setLogLevel(LogLevel.ERROR)
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
# ------------------------------------------------------------------------------
# Test Runners
# ------------------------------------------------------------------------------
# clique/snapshot_test.go(99): func TestClique(t *testing.T) {
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
proc runCliqueSnapshot(noisy = true; postProcessOk = false;
testIds = {0 .. 999}; skipIds = {0}-{0}) =
## Clique PoA Snapshot
## ::
## Tests that Clique signer voting is evaluated correctly for various
## simple and complex scenarios, as well as that a few special corner
## cases fail correctly.
##
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
let postProcessInfo = if postProcessOk: ", Transaction Finaliser Applied"
else: ", Without Finaliser"
suite &"Clique PoA Snapshot{postProcessInfo}":
var pool = newVoterPool()
setErrorLevel()
if noisy:
pool.noisy = true
setTraceLevel()
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
# clique/snapshot_test.go(379): for i, tt := range tests {
for voterSample in voterSamples.filterIt(it.id in testIds):
let tt = voterSample
test &"Snapshots {tt.id:2}: {tt.info.substr(0,50)}...":
pool.say "\n"
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
# Noisily skip this test
if tt.id in skipIds:
skip()
else:
# Assemble a chain of headers from the cast votes
# see clique/snapshot_test.go(407): config := *params.TestChainConfig
pool
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
.resetVoterChain(tt.signers, tt.epoch, tt.runBack)
# see clique/snapshot_test.go(425): for j, block := range blocks {
.appendVoter(tt.votes)
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
.commitVoterChain(postProcessOk)
# see clique/snapshot_test.go(477): if err != nil {
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
if tt.failure != cliqueNoError[0]:
# Note that clique/snapshot_test.go does not verify _here_ against
# the scheduled test error -- rather this voting error is supposed
# to happen earlier (processed at clique/snapshot_test.go(467)) when
# assembling the block chain (sounds counter intuitive to the author
# of this source file as the scheduled errors are _clique_ related).
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
check pool.failed[1][0] == tt.failure
else:
let
expected = tt.results.mapIt("@" & it).sorted
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
snapResult = pool.pp(pool.cliqueSigners).sorted
pool.say "*** snap state=", pool.pp(pool.snapshot,16)
pool.say " result=[", snapResult.join(",") & "]"
pool.say " expected=[", expected.join(",") & "]"
# Verify the final list of signers against the expected ones
check snapResult == expected
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
proc runCliqueSnapshot(noisy = true; postProcessOk = false; testId: int) =
noisy.runCliqueSnapshot(postProcessOk, testIds = {testId})
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
proc runGoerliReplay(noisy = true; showElapsed = false,
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
captureFile = goerliCapture,
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
startAtBlock = 0u64; stopAfterBlock = 0u64) =
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
var
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
pool = newVoterPool()
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
cache: array[groupReplayTransactions,(seq[BlockHeader],seq[BlockBody])]
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
cInx = 0
stoppedOk = false
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
let
fileInfo = captureFile.splitFile.name.split(".")[0]
filePath = captureFile.findFilePath
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
pool.verifyFrom = startAtBlock
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
setErrorLevel()
if noisy:
pool.noisy = true
setTraceLevel()
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
let stopThreshold = if stopAfterBlock == 0u64: uint64.high.u256
else: stopAfterBlock.u256
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
suite &"Replay Goerli chain from {fileInfo} capture":
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
for w in filePath.undumpBlocks:
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
if w[0][0].blockNumber == 0.u256:
# Verify Genesis
doAssert w[0][0] == pool.getBlockHeader(0.u256)
else:
# Condense in cache
cache[cInx] = w
cInx.inc
# Handy for partial tests
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
if stopThreshold < cache[cInx-1][0][0].blockNumber:
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
stoppedOk = true
break
# Run from cache if complete set
if cache.len <= cInx:
cInx = 0
let
first = cache[0][0][0].blockNumber
last = cache[^1][0][^1].blockNumber
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
blkRange = &"#{first}..#{last}"
info = if first <= startAtBlock.u256 and startAtBlock.u256 <= last:
&", verification #{startAtBlock}.."
else:
""
test &"Goerli Blocks {blkRange} ({cache.len} transactions{info})":
let start = getTime()
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
for (headers,bodies) in cache:
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
let addedPersistBlocks = pool.chain.persistBlocks(headers,bodies)
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
check addedPersistBlocks == ValidationResult.Ok
if addedPersistBlocks != ValidationResult.Ok: return
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
if showElapsed and startAtBlock.u256 <= last:
let
elpd = getTime() - start
info = &"{elpd.ppSecs:>7} {pool.cliqueSignersLen} {elpd.ppRow}"
echo &"\n elapsed {blkRange:<17} {info}"
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
# Rest from cache
if 0 < cInx:
let
first = cache[0][0][0].blockNumber
last = cache[cInx-1][0][^1].blockNumber
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
blkRange = &"#{first}..#{last}"
info = if first <= startAtBlock.u256 and startAtBlock.u256 <= last:
&", Verification #{startAtBlock}.."
else:
""
test &"Goerli Blocks {blkRange} ({cache.len} transactions{info})":
let start = getTime()
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
for (headers,bodies) in cache:
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
let addedPersistBlocks = pool.chain.persistBlocks(headers,bodies)
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
check addedPersistBlocks == ValidationResult.Ok
if addedPersistBlocks != ValidationResult.Ok: return
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
if showElapsed and startAtBlock.u256 <= last:
let
elpsd = getTime() - start
info = &"{elpsd.ppSecs:>7} {pool.cliqueSignersLen} {elpsd.ppRow}"
echo &"\n elapsed {blkRange:<17} {info}"
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
if stoppedOk:
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
test &"Runner stopped after reaching #{stopThreshold}":
discard
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
proc runGoerliBaybySteps(noisy = true;
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
captureFile = goerliCapture,
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
stopAfterBlock = 0u64) =
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
var
pool = newVoterPool()
stoppedOk = false
setErrorLevel()
if noisy:
pool.noisy = true
setTraceLevel()
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
let
fileInfo = captureFile.splitFile.name.split(".")[0]
filePath = captureFile.findFilePath
stopThreshold = if stopAfterBlock == 0u64: 20.u256
else: stopAfterBlock.u256
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
suite &"Replay Goerli chain from {fileInfo} capture, single blockwise":
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
for w in filePath.undumpBlocks:
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
if stoppedOk:
break
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
if w[0][0].blockNumber == 0.u256:
# Verify Genesis
doAssert w[0][0] == pool.getBlockHeader(0.u256)
else:
for n in 0 ..< w[0].len:
let
header = w[0][n]
body = w[1][n]
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
var
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
parents = w[0][0 ..< n]
test &"Goerli Block #{header.blockNumber} + {parents.len} parents":
check pool.chain.clique.cliqueSnapshot(header,parents).isOk
let addedPersistBlocks = pool.chain.persistBlocks(@[header],@[body])
check addedPersistBlocks == ValidationResult.Ok
if addedPersistBlocks != ValidationResult.Ok: return
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
# Handy for partial tests
if stopThreshold <= header.blockNumber:
stoppedOk = true
break
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
if stoppedOk:
test &"Runner stopped after reaching #{stopThreshold}":
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
discard
proc cliqueMiscTests() =
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
let
prvKeyFile = "private.key".findFilePath
suite "clique misc":
test "signer func":
let
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
engineSigner = "658bdf435d810c91414ec09147daa6db62406379"
privateKey = prvKeyFile
conf = makeConfig(@["--engine-signer:" & engineSigner, "--import-key:" & privateKey])
ctx = newEthContext()
check ctx.am.importPrivateKey(string conf.importKey).isOk()
check ctx.am.getAccount(conf.engineSigner).isOk()
proc signFunc(signer: EthAddress, message: openArray[byte]): Result[RawSignature, cstring] {.gcsafe.} =
let
hashData = keccakHash(message)
acc = ctx.am.getAccount(conf.engineSigner).tryGet()
rawSign = sign(acc.privateKey, SkMessage(hashData.data)).toRaw
ok(rawSign)
let signerFn: CliqueSignerFn = signFunc
var header: BlockHeader
header.extraData.setLen(EXTRA_VANITY)
header.extraData.add 0.byte.repeat(EXTRA_SEAL)
let signature = signerFn(conf.engineSigner, header.encodeSealHeader).get()
let extraLen = header.extraData.len
if EXTRA_SEAL < extraLen:
header.extraData.setLen(extraLen - EXTRA_SEAL)
header.extraData.add signature
let resAddr = ecRecover(header)
check resAddr.isOk
check resAddr.value == conf.engineSigner
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
# ------------------------------------------------------------------------------
# Main function(s)
# ------------------------------------------------------------------------------
proc cliqueMain*(noisy = defined(debug)) =
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
noisy.runCliqueSnapshot(true)
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
noisy.runCliqueSnapshot(false)
Feature/implement poa processing (#748) * re-shuffled Clique functions why: Due to the port from the go-sources, the interface logic is not optimal for nimbus. The main visible function is currently snapshot() and most of the _procurement_ of this function result has been moved to a sub-directory. * run eip-225 Clique test against p2p/chain.persistBlocks() why: Previously, loading the test block chains was fugdged with the purpose only to fill the database. As it is now clear how nimbus works on Goerli, the same can be achieved with a more realistic scenario. details: Eventually these tests will be pre-cursor to the reply tests for the Goerli chain supporting TDD approach with more simple cases. * fix exception annotations for executor module why: needed for exception tracking details: main annoyance are vmState methods (in state.nim) which potentially throw a base level Exception (a proc would only throws CatchableError) * split p2p/chain into sub-modules and fix exception annotations why: make space for implementing PoA stuff * provide over-loadable Clique PRNG why: There is a PRNG provided for generating reproducible number sequences. The functions which employ the PRNG to generate time slots were ported ported from the go-implementation. They are currently unused. * implement trusted signer assembly in p2p/chain.persistBlocks() details: * PoA processing moved there at the end of a transaction. Currently, there is no action (eg. transaction rollback) if this fails. * The unit tests with staged blocks work ok. In particular, there should be tests with to-be-rejected blocks. * TODO: 1.Optimise throughput/cache handling; 2.Verify headers * fix statement cast in pool.nim * added table features to LRU cache why: Clique uses the LRU cache using a mixture of volatile online items from the LRU cache and database checkpoints for hard synchronisation. For performance, Clique needs more table like features. details: First, last, and query key added, as well as efficient random delete added. Also key-item pair iterator added for debugging. * re-factored LRU snapshot caching why: Caching was sub-optimal (aka. bonkers) in that it skipped over memory caches in many cases and so mostly rebuild the snapshot from the last on-disk checkpoint. details; The LRU snapshot toValue() handler has been moved into the module clique_snapshot. This is for the fact that toValue() is not supposed to see the whole LRU cache database. So there must be a higher layer working with the the whole LRU cache and the on-disk checkpoint database. also: some clean up todo: The code still assumes that the block headers are valid in itself. This is particular important when an epoch header (aka re-sync header) is processed as it must contain the PoA result of all previous headers. So blocks need to be verified when they come in before used for PoA processing. * fix some snapshot cache fringe cases why: Must not index empty sequences in clique_snapshot module
2021-07-14 15:13:27 +00:00
noisy.runGoerliBaybySteps
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
noisy.runGoerliReplay(startAtBlock = 31100u64)
cliqueMiscTests()
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
when isMainModule:
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
let
skipIDs = {999}
# A new capture file can be generated using
# `test_clique/indiump.dumpGroupNl()`
# placed at the end of
# `p2p/chain/persist_blocks.persistBlocks()`.
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
captureFile = goerliCapture
#captureFile = "dump-stream.out.gz"
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
proc goerliReplay(noisy = true;
showElapsed = true;
captureFile = captureFile;
startAtBlock = 0u64;
stopAfterBlock = 0u64) =
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
runGoerliReplay(
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
noisy = noisy,
showElapsed = showElapsed,
captureFile = captureFile,
startAtBlock = startAtBlock,
stopAfterBlock = stopAfterBlock)
# local path is: nimbus-eth1/tests
let noisy = defined(debug)
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
Renamed source file clique_utils => clique_helpers (#762) * Renamed source file clique_utils => clique_helpers why: New name is more in line with other modules where local libraries are named similarly. * re-implemented PoA verification module as clique_verify.nim details: The verification code was ported from the go sources and provisionally stored in the clique_misc.nim source file. todo: Bring it to life. * re-design Snapshot descriptor as: ref object why: Avoids some copying descriptor objects details: The snapshot management in clique_snapshot.nim has been cleaned up. todo: There is a lot of unnecessary copying & sub-list manipulation of seq[BlockHeader] lists which needs to be simplified by managing index intervals. * optimised sequence handling for Clique/PoA why: To much ado about nothing details: * Working with shallow sequences inside PoA processing avoids unnecessary copying. * Using degenerate lists in the cliqueVerify() batch where only the parent (and no other ancestor) is needed. todo: Expose only functions that are needed, shallow sequences should be handles with care. * fix var-parameter function argument * Activate PoA engine -- currently proof of concept details: PoA engine is activated with newChain(extraValidation = true) applied to a PoA network. status and todo: The extraValidation flag on the Chain object can be set at a later state which allows to pre-load parts of the block chain without verification. Setting it later will only go back the block chain to the latest epoch checkpoint. This is inherent to the Clique protocol, needs testing though. PoA engine works in fine weather mode on Goerli replay. With the canonical eip-225 tests, there are quite a few fringe conditions that fail. These can easily fudged over to make things work but need some more work to understand and correct properly. * Make the last offending verification header available why: Makes some fringe case tests work. details: Within a failed transaction comprising several blocks, this feature help to identify the offending block if there was a PoA verification error. * Make PoA header verifier store the final snapshot why: The last snapshot needed by the verifier is the one of the parent but the list of authorised signer is derived from the current snapshot. So updating to the latest snapshot provides the latest signers list. details: Also, PoA processing has been implemented as transaction in persistBlocks() with Clique state rollback. Clique tests succeed now. * Avoiding double yields in iterator => replaced by template why: Tanks to Andri who observed it (see #762) * Calibrate logging interval and fix logging event detection why: Logging interval as copied from Go implementation was too large and needed re-calibration. Elapsed time calculation was bonkers, negative the wrong way round.
2021-07-21 13:31:52 +00:00
noisy.runCliqueSnapshot(true)
noisy.runCliqueSnapshot(false)
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
noisy.runGoerliBaybySteps
false.runGoerliReplay(startAtBlock = 31100u64)
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
Fearture/poa clique tuning (#765) * Provide API details: API is bundled via clique.nim. * Set extraValidation as default for PoA chains why: This triggers consensus verification and an update of the list of authorised signers. These signers are integral part of the PoA block chain. todo: Option argument to control validation for the nimbus binary. * Fix snapshot state block number why: Using sub-sequence here, so the len() function was wrong. * Optional start where block verification begins why: Can speed up time building loading initial parts of block chain. For PoA, this allows to prove & test that authorised signers can be (correctly) calculated starting at any point on the block chain. todo: On Goerli around blocks #193537..#197568, processing time increases disproportionally -- needs to be understand * For Clique test, get old grouping back (7 transactions per log entry) why: Forgot to change back after troubleshooting * Fix field/function/module-name misunderstanding why: Make compilation work * Use eth_types.blockHash() rather than utils.hash() in Clique modules why: Prefer lib module * Dissolve snapshot_misc.nim details: .. into clique_verify.nim (the other source file clique_unused.nim is inactive) * Hide unused AsyncLock in Clique descriptor details: Unused here but was part of the Go reference implementation * Remove fakeDiff flag from Clique descriptor details: This flag was a kludge in the Go reference implementation used for the canonical tests. The tests have been adapted so there is no need for the fakeDiff flag and its implementation. * Not observing minimum distance from epoch sync point why: For compiling PoA state, the go implementation will walk back to the epoch header with at least 90000 blocks apart from the current header in the absence of other synchronisation points. Here just the nearest epoch header is used. The assumption is that all the checkpoints before have been vetted already regardless of the current branch. details: The behaviour of using the nearest vs the minimum distance epoch is controlled by a flag and can be changed at run time. * Analysing processing time (patch adds some debugging/visualisation support) why: At the first half million blocks of the Goerli replay, blocks on the interval #194854..#196224 take exceptionally long to process, but not due to PoA processing. details: It turns out that much time is spent in p2p/excecutor.processBlock() where the elapsed transaction execution time is significantly greater for many of these blocks. Between the 1371 blocks #194854..#196224 there are 223 blocks with more than 1/2 seconds execution time whereas there are only 4 such blocks before and 13 such after this range up to #504192. * fix debugging symbol in clique_desc (causes CI failing) * Fixing canonical reference tests why: Two errors were introduced earlier but ovelooked: 1. "Remove fakeDiff flag .." patch was incomplete 2. "Not observing minimum distance .." introduced problem w/tests 23/24 details: Fixing 2. needed to revert the behaviour by setting the applySnapsMinBacklog flag for the Clique descriptor. Also a new test was added to lock the new behaviour. * Remove cruft why: Clique/PoA processing was intended to take place somewhere in executor/process_block.processBlock() but was decided later to run from chain/persist_block.persistBlock() instead. * Update API comment * ditto
2021-07-30 14:06:51 +00:00
#noisy.goerliReplay(startAtBlock = 31100u64)
#noisy.goerliReplay(startAtBlock = 194881u64, stopAfterBlock = 198912u64)
Redesign of BaseVMState descriptor (#923) * Redesign of BaseVMState descriptor why: BaseVMState provides an environment for executing transactions. The current descriptor also provides data that cannot generally be known within the execution environment, e.g. the total gasUsed which is available not before after all transactions have finished. Also, the BaseVMState constructor has been replaced by a constructor that does not need pre-initialised input of the account database. also: Previous constructor and some fields are provided with a deprecated annotation (producing a lot of noise.) * Replace legacy directives in production sources * Replace legacy directives in unit test sources * fix CI (missing premix update) * Remove legacy directives * chase CI problem * rebased * Re-introduce 'AccountsCache' constructor optimisation for 'BaseVmState' re-initialisation why: Constructing a new 'AccountsCache' descriptor can be avoided sometimes when the current state root is properly positioned already. Such a feature existed already as the update function 'initStateDB()' for the 'BaseChanDB' where the accounts cache was linked into this desctiptor. The function 'initStateDB()' was removed and re-implemented into the 'BaseVmState' constructor without optimisation. The old version was of restricted use as a wrong accounts cache state would unconditionally throw an exception rather than conceptually ask for a remedy. The optimised 'BaseVmState' re-initialisation has been implemented for the 'persistBlocks()' function. also: moved some test helpers to 'test/replay' folder * Remove unused & undocumented fields from Chain descriptor why: Reduces attack surface in general & improves reading the code.
2022-01-18 16:19:32 +00:00
cliqueMiscTests()
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
# ------------------------------------------------------------------------------
# End
Feature/goerli replay clique poa (#743) * extract unused clique/mining support into separate file why: mining is currently unsupported by nimbus * Replay first 51840 transactions from Goerli block chain why: Currently Goerli is loaded but the block headers are not verified. Replaying allows real data PoA development. details: Simple stupid gzipped dump/undump layer for debugging based on the zlib module (no nim-faststream support.) This is a replay running against p2p/chain.persistBlocks() where the data were captured from. * prepare stubs for PoA engine * split executor source into sup-modules why: make room for updates, clique integration should go into executor/update_poastate.nim * Simplify p2p/executor.processBlock() function prototype why: vmState argument always wraps basicChainDB * split processBlock() into sub-functions why: isolate the part where it will support clique/poa * provided additional processTransaction() function prototype without _fork_ argument why: with the exception of some tests, the _fork_ argument is always derived from the other prototype argument _vmState_ details: similar situation with makeReceipt() * provide new processBlock() version explicitly supporting PoA details: The new processBlock() version supporting PoA is the general one also supporting non-PoA networks, it needs an additional _Clique_ descriptor function argument for PoA state (if any.) The old processBlock() function without the _Clique_ descriptor argument retorns an error on PoA networgs (e.g. Goerli.) * re-implemented Clique descriptor as _ref object_ why: gives more flexibility when moving around the descriptor object details: also cleaned up a bit the clique sources * comments for clarifying handling of Clique/PoA state descriptor
2021-07-06 13:14:45 +00:00
# ------------------------------------------------------------------------------