status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/tests/test_attestation_pool.nim

571 lines
21 KiB
Nim
Raw Normal View History

naive attestation pool, in preparation of fork choice integration (#125) * move attestation pool to separate file * combine attestations lazily when needed * advance state when there's a gap while attesting * compile beacon node with optimizations - it's tooo slow right now * log when unable to keep up
2019-02-19 23:35:02 +00:00
# beacon_chain
remove some skipBlsValidation usage/handling in tests (#2258)
2021-01-22 13:29:04 +00:00
# Copyright (c) 2018-2021 Status Research & Development GmbH
naive attestation pool, in preparation of fork choice integration (#125) * move attestation pool to separate file * combine attestations lazily when needed * advance state when there's a gap while attesting * compile beacon node with optimizations - it's tooo slow right now * log when unable to keep up
2019-02-19 23:35:02 +00:00
# Licensed and distributed under either of
replace links to Apache and MIT licenses with HTTPS versions (#592)
2019-11-25 15:30:02 +00:00
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
naive attestation pool, in preparation of fork choice integration (#125) * move attestation pool to separate file * combine attestations lazily when needed * advance state when there's a gap while attesting * compile beacon node with optimizations - it's tooo slow right now * log when unable to keep up
2019-02-19 23:35:02 +00:00
# at your option. This file may not be copied, modified, or distributed except according to those terms.
remove UnusedImport spam (#549) * remove UnusedImport spam * re-add random import to beacon_node
2019-11-14 10:47:55 +00:00
{.used.}
naive attestation pool, in preparation of fork choice integration (#125) * move attestation pool to separate file * combine attestations lazily when needed * advance state when there's a gap while attesting * compile beacon node with optimizations - it's tooo slow right now * log when unable to keep up
2019-02-19 23:35:02 +00:00
import
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
std/sequtils,
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
# Status lib
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
unittest2,
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
chronicles, chronos,
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
stew/byteutils,
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
eth/keys,
# Internal
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
../beacon_chain/[beacon_node_types, extras],
../beacon_chain/gossip_processing/[gossip_validation],
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
../beacon_chain/fork_choice/[fork_choice_types, fork_choice],
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
../beacon_chain/consensus_object_pools/[
block_quarantine, blockchain_dag, block_clearance, attestation_pool],
../beacon_chain/ssz/merkleization,
../beacon_chain/spec/[crypto, datatypes, digest, validator, state_transition,
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
helpers, beaconstate, presets],
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
# Test utilities
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
./testutil, ./testdbutil, ./testblockutil
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
remove some skipBlsValidation usage/handling in tests (#2258)
2021-01-22 13:29:04 +00:00
func combine(tgt: var Attestation, src: Attestation) =
Bump nim-blscurve (#1491) * Bump BLSCurve * Use unified aggregation API * use new blscurve with unified aggregate API * bump * fix toRaw * replace state_sim combine with AggregateSignature * Fix 32-bit * Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag * change compilation switches to narrow down Linux issue * Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize * blscurve now disables both Loop and SLP vectorization * Add tests for the miracl/milagro fallback * Travis has max log size of 4MB * Test with Miracl in the finalization test * fix state_sim log level * Coment out the slow fallback tests
2020-08-15 17:33:58 +00:00
## Combine the signature and participation bitfield, with the assumption that
## the same data is being signed - if the signatures overlap, they are not
## combined.
doAssert tgt.data == src.data
# In a BLS aggregate signature, one needs to count how many times a
# particular public key has been added - since we use a single bit per key, we
# can only it once, thus we can never combine signatures that overlap already!
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
doAssert not tgt.aggregation_bits.overlaps(src.aggregation_bits)
Bump nim-blscurve (#1491) * Bump BLSCurve * Use unified aggregation API * use new blscurve with unified aggregate API * bump * fix toRaw * replace state_sim combine with AggregateSignature * Fix 32-bit * Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag * change compilation switches to narrow down Linux issue * Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize * blscurve now disables both Loop and SLP vectorization * Add tests for the miracl/milagro fallback * Travis has max log size of 4MB * Test with Miracl in the finalization test * fix state_sim log level * Coment out the slow fallback tests
2020-08-15 17:33:58 +00:00
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
tgt.aggregation_bits.incl(src.aggregation_bits)
var agg {.noInit.}: AggregateSignature
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
agg.init(tgt.signature.load().get())
agg.aggregate(src.signature.load.get())
tgt.signature = agg.finish().toValidatorSig()
Bump nim-blscurve (#1491) * Bump BLSCurve * Use unified aggregation API * use new blscurve with unified aggregate API * bump * fix toRaw * replace state_sim combine with AggregateSignature * Fix 32-bit * Fix 32-bit for real and test deactivating ccache for fno-tree-lopp-vectorize flag * change compilation switches to narrow down Linux issue * Use -fno-tree-vectorize to disable both tree-loop-vectorize and tree-slp-vectorize * blscurve now disables both Loop and SLP vectorization * Add tests for the miracl/milagro fallback * Travis has max log size of 4MB * Test with Miracl in the finalization test * fix state_sim log level * Coment out the slow fallback tests
2020-08-15 17:33:58 +00:00
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
func loadSig(a: Attestation): CookedSig =
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
a.signature.load.get()
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
Move pruning out of latency critical path (#2384) * Deferred DAG and fork choice pruning * fixup * Address https://github.com/status-im/nimbus-eth2/pull/2384/files#r589448448, rely only on onSLotEnd for state pruning * no need to store needPruning in the data structure * lastPrunePoint is updated in pruning proc * Split eager and LazyPruning * enforce pruning in updateHead
2021-03-09 14:36:17 +00:00
proc pruneAtFinalization(dag: ChainDAGRef, attPool: AttestationPool) =
if dag.needStateCachesAndForkChoicePruning():
dag.pruneStateCachesDAG()
# pool[].prune() # We test logic without attestation pool / fork choice pruning
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
suite "Attestation pool processing" & preset():
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
## For now just test that we can compile and execute block processing with
## mock data.
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
setup:
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
# Genesis state that results in 6 members per committee
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
var
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
chainDag = init(ChainDAGRef, defaultRuntimePreset, makeTestDB(SLOTS_PER_EPOCH * 6))
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
quarantine = QuarantineRef.init(keys.newRng())
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
pool = newClone(AttestationPool.init(chainDag, quarantine))
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
state = newClone(chainDag.headState)
stop discarding future epochs; remove a StateCache() construction (#1610) * stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots() * update nbench to pass StateCache to process_slots()
2020-09-07 15:04:33 +00:00
cache = StateCache()
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
rewards: RewardInfo
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
# Slot 0 is a finalized slot - won't be making attestations for it..
check that process_slots() succeeds
2020-05-19 15:46:29 +00:00
check:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slots(state.data, getStateField(state, slot) + 1, cache, rewards)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Can add and retrieve simple attestations" & preset():
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
# Create an attestation for slot 1!
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
bc0 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
attestation = makeAttestation(
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
state.data.data, state.blck.root, bc0[0], cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
pool[].addAttestation(
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
attestation, @[bc0[0]], attestation.loadSig,
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
attestation.data.slot)
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
check that process_slots() succeeds
2020-05-19 15:46:29 +00:00
check:
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
# Added attestation, should get it back
toSeq(pool[].attestations(none(Slot), none(CommitteeIndex))) ==
@[attestation]
toSeq(pool[].attestations(
some(attestation.data.slot), none(CommitteeIndex))) == @[attestation]
toSeq(pool[].attestations(
some(attestation.data.slot), some(attestation.data.index.CommitteeIndex))) ==
@[attestation]
toSeq(pool[].attestations(none(Slot), some(attestation.data.index.CommitteeIndex))) ==
@[attestation]
toSeq(pool[].attestations(some(
attestation.data.slot + 1), none(CommitteeIndex))) == []
toSeq(pool[].attestations(
none(Slot), some(CommitteeIndex(attestation.data.index + 1)))) == []
process_slots(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
getStateField(state, slot) + MIN_ATTESTATION_INCLUSION_DELAY, cache,
rewards)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
fix some todo (#1645) * remove some superfluous gcsafes * remove getTailState (unused) * don't store old epochrefs in blocks * document attestation pool a bit * remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
attestations.len == 1
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
pool[].getAggregatedAttestation(1.Slot, 0.CommitteeIndex).isSome()
let
root1 = addTestBlock(
state.data, state.blck.root,
cache, attestations = attestations, nextSlot = false).root
bc1 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
att1 = makeAttestation(
state.data.data, root1, bc1[0], cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
check:
process_slots(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
getStateField(state, slot) + MIN_ATTESTATION_INCLUSION_DELAY, cache,
rewards)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
check:
# shouldn't include already-included attestations
pool[].getAttestationsForBlock(state.data.data, cache) == []
pool[].addAttestation(
att1, @[bc1[0]], att1.loadSig, att1.data.slot)
check:
# but new ones should go in
pool[].getAttestationsForBlock(state.data.data, cache).len() == 1
let
att2 = makeAttestation(
state.data.data, root1, bc1[1], cache)
pool[].addAttestation(
att2, @[bc1[1]], att2.loadSig, att2.data.slot)
let
combined = pool[].getAttestationsForBlock(state.data.data, cache)
check:
# New attestations should be combined with old attestations
combined.len() == 1
combined[0].aggregation_bits.countOnes() == 2
pool[].addAttestation(
combined[0], @[bc1[1], bc1[0]], combined[0].loadSig, combined[0].data.slot)
check:
# readding the combined attestation shouldn't have an effect
pool[].getAttestationsForBlock(state.data.data, cache).len() == 1
let
# Someone votes for a different root
att3 = makeAttestation(state.data.data, Eth2Digest(), bc1[2], cache)
pool[].addAttestation(
att3, @[bc1[2]], att3.loadSig, att3.data.slot)
check:
# We should now get both attestations for the block, but the aggregate
# should be the one with the most votes
pool[].getAttestationsForBlock(state.data.data, cache).len() == 2
pool[].getAggregatedAttestation(2.Slot, 0.CommitteeIndex).
get().aggregation_bits.countOnes() == 2
pool[].getAggregatedAttestation(2.Slot, hash_tree_root(att2.data)).
get().aggregation_bits.countOnes() == 2
let
# Someone votes for a different root
att4 = makeAttestation(state.data.data, Eth2Digest(), bc1[2], cache)
pool[].addAttestation(
att4, @[bc1[2]], att3.loadSig, att3.data.slot)
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Working with aggregates" & preset():
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
let
# Create an attestation for slot 1!
bc0 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
var
att0 = makeAttestation(state.data.data, state.blck.root, bc0[0], cache)
att0x = att0
att1 = makeAttestation(state.data.data, state.blck.root, bc0[1], cache)
att2 = makeAttestation(state.data.data, state.blck.root, bc0[2], cache)
att3 = makeAttestation(state.data.data, state.blck.root, bc0[3], cache)
# Both attestations include member 2 but neither is a subset of the other
att0.combine(att2)
att1.combine(att2)
pool[].addAttestation(att0, @[bc0[0], bc0[2]], att0.loadSig, att0.data.slot)
pool[].addAttestation(att1, @[bc0[1], bc0[2]], att1.loadSig, att1.data.slot)
check:
process_slots(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data,
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
getStateField(state, slot) + MIN_ATTESTATION_INCLUSION_DELAY, cache,
rewards)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
check:
pool[].getAttestationsForBlock(state.data.data, cache).len() == 2
# Can get either aggregate here, random!
pool[].getAggregatedAttestation(1.Slot, 0.CommitteeIndex).isSome()
# Add in attestation 3 - both aggregates should now have it added
pool[].addAttestation(att3, @[bc0[3]], att3.loadSig, att3.data.slot)
block:
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
check:
attestations.len() == 2
attestations[0].aggregation_bits.countOnes() == 3
# Can get either aggregate here, random!
pool[].getAggregatedAttestation(1.Slot, 0.CommitteeIndex).isSome()
# Add in attestation 0 as single - attestation 1 is now a superset of the
# aggregates in the pool, so everything else should be removed
pool[].addAttestation(att0x, @[bc0[0]], att0x.loadSig, att0x.data.slot)
block:
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
check:
attestations.len() == 1
attestations[0].aggregation_bits.countOnes() == 4
pool[].getAggregatedAttestation(1.Slot, 0.CommitteeIndex).isSome()
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Everyone voting for something different" & preset():
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
var attestations: int
for i in 0..<SLOTS_PER_EPOCH:
var root: Eth2Digest
root.data[0..<8] = toBytesBE(i.uint64)
let
bc0 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
for j in 0..<bc0.len():
root.data[8..<16] = toBytesBE(j.uint64)
var att = makeAttestation(state.data.data, root, bc0[j], cache)
pool[].addAttestation(att, @[bc0[j]], att.loadSig, att.data.slot)
inc attestations
check:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slots(state.data, getStateField(state, slot) + 1, cache,
rewards)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
doAssert attestations.uint64 > MAX_ATTESTATIONS,
"6*SLOTS_PER_EPOCH validators > 128 mainnet MAX_ATTESTATIONS"
check:
# Fill block with attestations
pool[].getAttestationsForBlock(state.data.data, cache).lenu64() ==
MAX_ATTESTATIONS
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
pool[].getAggregatedAttestation(
getStateField(state, slot) - 1, 0.CommitteeIndex).isSome()
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Attestations may arrive in any order" & preset():
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
# Create an attestation for slot 1!
bc0 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
attestation0 = makeAttestation(
state.data.data, state.blck.root, bc0[0], cache)
check that process_slots() succeeds
2020-05-19 15:46:29 +00:00
check:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slots(state.data, getStateField(state, slot) + 1, cache, rewards)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
bc1 = get_beacon_committee(state.data.data,
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
getStateField(state, slot), 0.CommitteeIndex, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
attestation1 = makeAttestation(
state.data.data, state.blck.root, bc1[0], cache)
# test reverse order
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation1, @[bc1[0]], attestation1.loadSig, attestation1.data.slot)
pool[].addAttestation(
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
attestation0, @[bc0[0]], attestation0.loadSig, attestation0.data.slot)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
stop discarding future epochs; remove a StateCache() construction (#1610) * stop discarding non-existent future epochs during epoch state transitions; remove a pointless StateCache() construction in advance_slots() * update nbench to pass StateCache to process_slots()
2020-09-07 15:04:33 +00:00
discard process_slots(
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
state.data, MIN_ATTESTATION_INCLUSION_DELAY.Slot + 1, cache, rewards)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
fix some todo (#1645) * remove some superfluous gcsafes * remove getTailState (unused) * don't store old epochrefs in blocks * document attestation pool a bit * remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
attestations.len == 1
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Attestations should be combined" & preset():
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
# Create an attestation for slot 1!
bc0 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
attestation0 = makeAttestation(
state.data.data, state.blck.root, bc0[0], cache)
attestation1 = makeAttestation(
state.data.data, state.blck.root, bc0[1], cache)
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation0, @[bc0[0]], attestation0.loadSig, attestation0.data.slot)
pool[].addAttestation(
attestation1, @[bc0[1]], attestation1.loadSig, attestation1.data.slot)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check that process_slots() succeeds
2020-05-19 15:46:29 +00:00
check:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slots(
state.data, MIN_ATTESTATION_INCLUSION_DELAY.Slot + 1, cache, rewards)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
fix some todo (#1645) * remove some superfluous gcsafes * remove getTailState (unused) * don't store old epochrefs in blocks * document attestation pool a bit * remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
attestations.len == 1
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Attestations may overlap, bigger first" & preset():
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
var
# Create an attestation for slot 1!
bc0 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), 0.CommitteeIndex, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
attestation0 = makeAttestation(
state.data.data, state.blck.root, bc0[0], cache)
attestation1 = makeAttestation(
state.data.data, state.blck.root, bc0[1], cache)
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
remove some skipBlsValidation usage/handling in tests (#2258)
2021-01-22 13:29:04 +00:00
attestation0.combine(attestation1)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation0, @[bc0[0]], attestation0.loadSig, attestation0.data.slot)
pool[].addAttestation(
attestation1, @[bc0[1]], attestation1.loadSig, attestation1.data.slot)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
check that process_slots() succeeds
2020-05-19 15:46:29 +00:00
check:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slots(
state.data, MIN_ATTESTATION_INCLUSION_DELAY.Slot + 1, cache, rewards)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
fix some todo (#1645) * remove some superfluous gcsafes * remove getTailState (unused) * don't store old epochrefs in blocks * document attestation pool a bit * remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
attestations.len == 1
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Attestations may overlap, smaller first" & preset():
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
var
# Create an attestation for slot 1!
bc0 = get_beacon_committee(state.data.data,
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
getStateField(state, slot), 0.CommitteeIndex, cache)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
attestation0 = makeAttestation(
state.data.data, state.blck.root, bc0[0], cache)
attestation1 = makeAttestation(
state.data.data, state.blck.root, bc0[1], cache)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
remove some skipBlsValidation usage/handling in tests (#2258)
2021-01-22 13:29:04 +00:00
attestation0.combine(attestation1)
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation1, @[bc0[1]], attestation1.loadSig, attestation1.data.slot)
pool[].addAttestation(
attestation0, @[bc0[0]], attestation0.loadSig, attestation0.data.slot)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
check that process_slots() succeeds
2020-05-19 15:46:29 +00:00
check:
add attestation stats tool to ncli_db (#2539) This also makes future efforts to provide metrics and logs for attestation efficiency easier * Export rewards from epoch transition * Use less memory for reward calculation (bool -> set[enum], field alignment) * Reuse reward memory when replaying, avoiding spike * Allow replaying any range in ncli_db benchmark
2021-05-07 11:36:21 +00:00
process_slots(
state.data, MIN_ATTESTATION_INCLUSION_DELAY.Slot + 1, cache, rewards)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
fix some todo (#1645) * remove some superfluous gcsafes * remove getTailState (unused) * don't store old epochrefs in blocks * document attestation pool a bit * remove `pcs =` cruft from log
2020-09-14 14:50:03 +00:00
let attestations = pool[].getAttestationsForBlock(state.data.data, cache)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
attestations.len == 1
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Fork choice returns latest block with no attestations":
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
b1 = addTestBlock(state.data, chainDag.tail.root, cache)
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
b1Add = chainDag.addRawBlock(quarantine, b1) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head = pool[].selectHead(b1Add[].slot)
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
head == b1Add[]
disable block/attestation pool tests in mainnet
2019-12-19 14:13:35 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
b2 = addTestBlock(state.data, b1.root, cache)
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
b2Add = chainDag.addRawBlock(quarantine, b2) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head2 = pool[].selectHead(b2Add[].slot)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
head2 == b2Add[]
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Fork choice returns block with attestation":
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
b10 = makeTestBlock(state.data, chainDag.tail.root, cache)
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
b10Add = chainDag.addRawBlock(quarantine, b10) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head = pool[].selectHead(b10Add[].slot)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
head == b10Add[]
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
b11 = makeTestBlock(state.data, chainDag.tail.root, cache,
Add an option for graffiti customization
2020-06-29 17:30:19 +00:00
graffiti = GraffitiBytes [1'u8, 0, 0, 0 ,0 ,0 ,0 ,0 ,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
)
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
b11Add = chainDag.addRawBlock(quarantine, b11) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Move some attestation/block logic out of beacon node * state data cache in block pool * keep head state around * more attestation logic in attestation pool * first fork choice tests (!) * fix fork choice (it's still likely broken / out of date)
2019-12-19 13:02:28 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
bc1 = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot) - 1, 1.CommitteeIndex, cache)
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
attestation0 = makeAttestation(state.data.data, b10.root, bc1[0], cache)
Add support for dealing with overlapping attestations (#262)
2019-06-03 08:26:38 +00:00
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation0, @[bc1[0]], attestation0.loadSig, attestation0.data.slot)
Add support for dealing with overlapping attestations (#262)
2019-06-03 08:26:38 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head2 = pool[].selectHead(b10Add[].slot)
Add support for dealing with overlapping attestations (#262)
2019-06-03 08:26:38 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
# Single vote for b10 and no votes for b11
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
head2 == b10Add[]
Add support for dealing with overlapping attestations (#262)
2019-06-03 08:26:38 +00:00
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
let
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
attestation1 = makeAttestation(state.data.data, b11.root, bc1[1], cache)
attestation2 = makeAttestation(state.data.data, b11.root, bc1[2], cache)
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation1, @[bc1[1]], attestation1.loadSig, attestation1.data.slot)
Add support for dealing with overlapping attestations (#262)
2019-06-03 08:26:38 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head3 = pool[].selectHead(b10Add[].slot)
let bigger = if b11.root.data < b10.root.data: b10Add else: b11Add
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
# Ties broken lexicographically in spec -> ?
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
head3 == bigger[]
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
only deserialize attestation and aggregation gossiped signatures once (#2472) * only deserialize attestation and aggregation gossiped signatures once * re-indent some aggregate checks into block scope * spelling * remove debugging assertion * put part of gossip validation back into block context * attestation pool test signature loading isn't so unsafe, and exportRaw isn't free * remove more development doAsserts; don't exportRaw in loops
2021-04-09 12:59:24 +00:00
pool[].addAttestation(
attestation2, @[bc1[2]], attestation2.loadSig, attestation2.data.slot)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head4 = pool[].selectHead(b11Add[].slot)
re-enable mainnet attestation pool tests (#946) * re-enable mainnet attestation pool tests; since https://github.com/status-im/nim-beacon-chain/pull/930 double CI state sim validators * bump deposits spec ref to v0.11.1
2020-04-29 11:44:07 +00:00
check:
# Two votes for b11
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
head4 == b11Add[]
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Trying to add a block twice tags the second as an error":
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
var cache = StateCache()
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
let
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
b10 = makeTestBlock(state.data, chainDag.tail.root, cache)
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
b10Add = chainDag.addRawBlock(quarantine, b10) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head = pool[].selectHead(b10Add[].slot)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
check:
head == b10Add[]
# -------------------------------------------------------------
# Add back the old block to ensure we have a duplicate error
let b10_clone = b10 # Assumes deep copy
renamed CandidateChains to ChainDagRef and made the Quarantine type a ref type so there is a single instance in the beacon node (#1407)
2020-07-31 14:49:06 +00:00
let b10Add_clone = chainDag.addRawBlock(quarantine, b10_clone) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
bump libp2p; integrate pubsub.ValidationResult into extended validation (#1893)
2020-10-20 12:31:20 +00:00
doAssert: b10Add_clone.error == (ValidationResult.Ignore, Duplicate)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
test "Trying to add a duplicate block from an old pruned epoch is tagged as an error":
Fork choice - almost free pruning - fix #1534 (#1535) * initial - cheaper pruning - addresses #1534 * Pass tests: update offset when pruning, proper handling of pruned parents * Use options instead of nil for nilable newHead (finalization passing but rootcause not solved) * First line of defense against stackoverflow in tests * Fix compute_delta offset after pruning * Rebase fix - medalla ready * Remove Option[BlockRef]
2020-08-26 15:23:34 +00:00
# Note: very sensitive to stack usage
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
chainDag.updateFlags.incl {skipBLSValidation}
cache beacon committee size calculation (#1363) * cache beacon committee size calculation this fixes a bug in get_validator_churn_limit as well * fix * make committee counts consistently uint64 mixing feels like the worst of the two worlds
2020-07-23 17:01:07 +00:00
var cache = StateCache()
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
let
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
b10 = addTestBlock(state.data, chainDag.tail.root, cache)
b10Add = chainDag.addRawBlock(quarantine, b10) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head = pool[].selectHead(b10Add[].slot)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
doAssert: head == b10Add[]
# -------------------------------------------------------------
let b10_clone = b10 # Assumes deep copy
# -------------------------------------------------------------
# Pass an epoch
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
var block_root = b10.root
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
var attestations: seq[Attestation]
for epoch in 0 ..< 5:
let start_slot = compute_start_slot_at_epoch(Epoch epoch)
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let committees_per_slot =
v0.12.2 beacon chain protocol update (#1378)
2020-07-27 10:59:57 +00:00
get_committee_count_per_slot(state.data.data, Epoch epoch, cache)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
for slot in start_slot ..< start_slot + SLOTS_PER_EPOCH:
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
let new_block = addTestBlock(
state.data, block_root, cache, attestations = attestations)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
cache block hash (#1329) hash_tree_root was turning up when running beacon_node, turns out to be repeated hash_tree_root invocations - this pr brings them back down to normal. this PR caches the root of a block in the SignedBeaconBlock object - this has the potential downside that even invalid blocks will be hashed (as part of deserialization) - later, one could imagine delaying this until checks have passed there's also some cleanup of the `cat=` logs which were applied randomly and haphazardly, and to a large degree are duplicated by other information in the log statements - in particular, topics fulfill the same role
2020-07-16 13:16:51 +00:00
block_root = new_block.root
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
let blockRef = chainDag.addRawBlock(quarantine, new_block) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
let head = pool[].selectHead(blockRef[].slot)
Revamp attestation pool This is a revamp of the attestation pool that cleans up several aspects of attestation processing as the network grows larger and block space becomes more precious. The aim is to better exploit the divide between attestation subnets and aggregations by keeping the two kinds separate until it's time to either produce a block or aggregate. This means we're no longer eagerly combining single-vote attestations, but rather wait until the last moment, and then try to add singles to all aggregates, including those coming from the network. Importantly, the branch improves on poor aggregate quality and poor attestation packing in cases where block space is running out. A basic greed scoring mechanism is used to select attestations for blocks - attestations are added based on how much many new votes they bring to the table. * Collect single-vote attestations separately and store these until it's time to make aggregates * Create aggregates based on single-vote attestations * Select _best_ aggregate rather than _first_ aggregate when on aggregation duty * Top up all aggregates with singles when it's time make the attestation cut, thus improving the chances of grabbing the best aggregates out there * Improve aggregation test coverage * Improve bitseq operations * Simplify aggregate signature creation * Make attestation cache temporary instead of storing it in attestation pool - most of the time, blocks are not being produced, no need to keep the data around * Remove redundant aggregate storage that was used only for RPC * Use tables to avoid some linear seeks when looking up attestation data * Fix long cleanup on large slot jumps * Avoid some pointers * Speed up iterating all attestations for a slot (fixes #2490)
2021-04-12 20:25:09 +00:00
doAssert: head == blockRef[]
bound block quarantine size (#1564) * bound block quarantine size * add additional logging for block quarantining * re-add quarantine.add() call * remove pre-finalization blocks; add logging for full quarantine * clear quarantine on chain reorganization * update block_sim and tests * update test_attestation_pool
2020-08-31 09:00:38 +00:00
chainDag.updateHead(head, quarantine)
Move pruning out of latency critical path (#2384) * Deferred DAG and fork choice pruning * fixup * Address https://github.com/status-im/nimbus-eth2/pull/2384/files#r589448448, rely only on onSLotEnd for state pruning * no need to store needPruning in the data structure * lastPrunePoint is updated in pruning proc * Split eager and LazyPruning * enforce pruning in updateHead
2021-03-09 14:36:17 +00:00
pruneAtFinalization(chainDag, pool[])
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
attestations.setlen(0)
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
for index in 0'u64 ..< committees_per_slot:
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
let committee = get_beacon_committee(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot), index.CommitteeIndex,
cache)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
# Create a bitfield filled with the given count per attestation,
# exactly on the right-most part of the committee field.
var aggregation_bits = init(CommitteeValidatorsBits, committee.len)
for v in 0 ..< committee.len * 2 div 3 + 1:
aggregation_bits[v] = true
attestations.add Attestation(
aggregation_bits: aggregation_bits,
data: makeAttestationData(
more abstraction over BeaconState (#2509) * more abstraction over BeaconState * use HashedBeaconState copy of htr
2021-04-16 08:49:37 +00:00
state.data.data, getStateField(state, slot),
Datatype cleanup (#1953) * clear up spec todo * test fix * remove unnecessary toSszType * type * one more
2020-11-04 21:52:47 +00:00
index.CommitteeIndex, blockroot)
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
# signature: ValidatorSig()
)
restore EpochRef and flush statecaches on epoch transitions (#1312) * restore EpochRef and flush statecaches on epoch transitions * more targeted cache invalidation * remove get_empty_per_epoch_cache(); implement simpler but still faster get_beacon_proposer_index()/compute_proposer_index() approach; add some abstraction layer for accessing the shuffled validator indices cache * reduce integer type conversions * remove most of rest of integer type conversion in compute_proposer_index()
2020-07-15 10:44:18 +00:00
cache = StateCache()
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
# -------------------------------------------------------------
# Prune
removed the BlockPool type and all of the proxy functions around it (#1401) * removed the BlockPool type and all of the proxy functions around it - passing the chain DAG and the quarantine explicitly where appropriately - they don't need to be bundled in a type * fixed the build after the rebase
2020-07-30 19:18:17 +00:00
doAssert: chainDag.finalizedHead.slot != 0
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
Use fork v2 (#1358) * fork choice fixes, round 3 * introduce checkpoint tracker * split out fork choice backend that is independent of dag * correctly update best checkpoint to use for head selection * correctly consider wall clock when processing attestations * preload head history only (only one history is loaded from database anyway) * love the DAG * switch to fork choice v2 also remove BlockRef.children * fix
2020-07-25 19:41:12 +00:00
pool[].prune()
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
doAssert: b10.root notin pool.forkChoice.backend
Dual headed fork choice [Revolution] (#1238) * Dual headed fork choice * fix finalizedEpoch not moving * reduce fork choice verbosity * Add failing tests due to pruning * Properly handle duplicate blocks in sync * test_block_pool also add a test for duplicate blocks * comments addressing review * Fix fork choice v2, was missing integrating block proposed * remove a spurious debug writeStackTrace * update block_sim * Use OrderedTable to ensure that we always load parents before children in fork choice * Load the DAG data in fork choice at init if there is some (can sync witti) * Cluster of quarantined blocks were not properly added to the fork choice * Workaround async gcsafe warnings * Update blockpoool tests * Do the callback before clearing the quarantine * Revert OrderedTable, implement topological sort of DAG, allow forkChoice to be initialized from arbitrary finalized heads * Make it work with latest devel - Altona readyness * Add a recovery mechanism when forkchoice desyncs with blockpool * add the current problematic node to the stack * Fix rebase indentation bug (but still producing invalid block) * Fix cache at epoch boundaries and lateBlock addition
2020-07-09 09:29:32 +00:00
# Add back the old block to ensure we have a duplicate error
implement clock disparity for attestation validation (#1568) This implements disparity, resolving a part of https://github.com/status-im/nim-beacon-chain/issues/1367 * make BeaconTime a duration for fractional seconds * factor out attestation/aggregate validation * simplify recording of queued attestations * simplify attestation signature check * fix blocks_received metric * add some trivial validation tests * remove unresolved attestation table - attestations for unknown blocks are dropped instead (cannot verify their signature)
2020-08-27 07:34:12 +00:00
let b10Add_clone = chainDag.addRawBlock(quarantine, b10_clone) do (
Block validation flow v2 + Batch (serial) sig verification (#2250) * bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561819858 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828025 * onBlockAdded callback should use TrustedSignedBeaconBlock https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561837261 * address https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561828946 * Use the application RNG: https://github.com/status-im/nimbus-eth2/pull/2250#discussion_r561815336 * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded
2021-01-25 18:45:48 +00:00
blckRef: BlockRef, signedBlock: TrustedSignedBeaconBlock,
rework epoch cache referencing * collect all epochrefs in specific blocks to make them easier to find and to avoid lots of small seqs * reuse validator key databases more aggressively by comparing keys * make state cache available from within `withState` * make epochRef available from within onBlockAdded callback * integrate getEpochInfo into block resolution and epoch ref logic such that epochrefs are created when blocks are added to pool or lazily when needed by a getEpochRef * fill state cache better from EpochRef, speeding up replay and validation * store epochRef in specific blocks to make them easier to find and reuse * fix database corruption when state is saved while replaying quarantine * replay slots fully from block pool before processing state * compare bls values more smartly * store epoch state without block applied in database - it's recommended to resync the node! this branch will drastically speed up processing in times of long non-finality, as well as cut memory usage by 10x during the recent medalla madness.
2020-08-18 20:29:33 +00:00
epochRef: EpochRef, state: HashedBeaconState):
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
# Callback add to fork choice if valid
use epoch ref to update fork choice this dramatically speeds up startup in long periods of non-finality
2020-08-03 18:39:43 +00:00
pool[].addForkChoice(epochRef, blckRef, signedBlock.message, blckRef.slot)
Fork choice fixes (#1350) * remove cruft * reenable fork choice and fix several issues * in addForkChoice_v2, the `.error` field would be accessed even when Result is ok * remove workaround for invalid block structure in fork choice * fix `tmpState` being used recursively in callback, causing state corruption while processing attestation * fix block callback being called twice per block * pass state to callback to avoid unnecessary rewinding * enable head select, fix another bug * never use `get` without `isOk` * log nil blockref in case blockref is nil * add missing error checking * use correct epoch when updating attestation message
2020-07-22 09:42:55 +00:00
bump libp2p; integrate pubsub.ValidationResult into extended validation (#1893)
2020-10-20 12:31:20 +00:00
doAssert: b10Add_clone.error == (ValidationResult.Ignore, Duplicate)