status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/tests/test_gossip_validation.nim

291 lines
11 KiB
Nim
Raw Permalink Normal View History

remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# beacon_chain
refactor batch validation not to require genesis_validators_root each time (#4640)
2023-02-20 08:26:22 +00:00
# Copyright (c) 2018-2023 Status Research & Development GmbH
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
{.used.}
import
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
# Standard library
std/sequtils,
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# Status lib
unittest2,
remove unused imports in tests/ (#3713)
2022-06-07 17:05:06 +00:00
chronos,
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
taskpools,
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# Internal
import cleanup (#2997) * import cleanup ...and remove some unused types * add random imports * more imports
2021-10-19 14:09:26 +00:00
../beacon_chain/[beacon_clock],
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
../beacon_chain/gossip_processing/[gossip_validation, batch_validation],
remove unused imports in tests/ (#3713)
2022-06-07 17:05:06 +00:00
../beacon_chain/fork_choice/fork_choice,
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
../beacon_chain/consensus_object_pools/[
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
block_quarantine, blockchain_dag, block_clearance, attestation_pool,
sync_committee_msg_pool],
../beacon_chain/spec/datatypes/[phase0, altair],
use correct root in `test_gossip_validation` (#4951) Use block root not state root for sync committee message in test.
2023-05-14 14:18:50 +00:00
../beacon_chain/spec/[
beaconstate, state_transition, helpers, network, validator],
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
../beacon_chain/validators/validator_pool,
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# Test utilities
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
./testutil, ./testdbutil, ./testblockutil
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
proc pruneAtFinalization(dag: ChainDAGRef, attPool: AttestationPool) =
if dag.needStateCachesAndForkChoicePruning():
dag.pruneStateCachesDAG()
# pool[].prune() # We test logic without att_1_0 pool / fork choice pruning
Introduce unittest2 and junit reports (#2522) * Introduce unittest2 and junit reports * fix XML path * don't combine multiple CI runs * fixup * public combined report also Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2021-04-28 16:41:02 +00:00
suite "Gossip validation " & preset():
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
setup:
# Genesis state that results in 3 members per committee
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
let rng = HmacDrbgContext.new()
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
var
Validator monitoring (#2925) Validator monitoring based on and mostly compatible with the implementation in Lighthouse - tracks additional logs and metrics for specified validators so as to stay on top on performance. The implementation works more or less the following way: * Validator pubkeys are singled out for monitoring - these can be running on the node or not * For every action that the validator takes, we record steps in the process such as messages being seen on the network or published in the API * When the dust settles at the end of an epoch, we report the information from one epoch before that, which coincides with the balances being updated - this is a tradeoff between being correct (waiting for finalization) and providing relevant information in a timely manner)
2021-12-20 19:20:31 +00:00
validatorMonitor = newClone(ValidatorMonitor.init())
dag = init(
ChainDAGRef, defaultRuntimeConfig, makeTestDB(SLOTS_PER_EPOCH * 3),
validatorMonitor, {})
Parallel attestation verification (#2718) * Add parallel attestation verification * Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update) * bump nim-blscurve * Debug info for failing eth2 test vectors * remove submodule eth2-testnets * verbose debugging of make failure on Windows (libbacktrace?) * Remove CI debug mode * initialization convention * Fix new altair tests
2021-09-17 00:13:52 +00:00
taskpool = Taskpool.new()
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
verifier = BatchVerifier(rng: rng, taskpool: taskpool)
move quarantine outside of chaindag (#3124) * move quarantine outside of chaindag The quarantine has been part of the ChainDAG for the longest time, but this design has a few issues: * the function in which blocks are verified and added to the dag becomes reentrant and therefore difficult to reason about - we're currently using a stateful flag to work around it * quarantined blocks bypass the processing queue leading to a processing stampede * the quarantine flow is unsuitable for orphaned attestations - these should also should be quarantined eventually Instead of processing the quarantine inside ChainDAG, this PR moves re-queueing to `block_processor` which already is responsible for dealing with follow-up work when a block is added to the dag This sets the stage for keeping attestations in the quarantine as well. Also: * make `BlockError` `{.pure.}` * avoid use of `ValidationResult` in block clearance (that's for gossip)
2021-12-06 09:49:01 +00:00
quarantine = newClone(Quarantine.init())
singe validator key cache Instead of keeping a validator key list per EpochRef, this PR introduces a single shared validator key list in ChainDAG, and cleans up some other ChainDAG and key-related issues. The PR does not introduce the validator key list in the state transition - this is because we batch-check all signatures before entering the spec code, thus the spec code never hits the cache. A future refactor should _probably_ remove the threadvar altogether. There's a few other small fixes in here that make the flow easier to read: * fix `var ChainDAGRef` -> `ChainDAGRef` * fix `var QuarantineRef` -> `QuarantineRef` * consistent `dag` variable name * avoid using threadvar pubkey cache in most cases * better error messages in batch signature checking
2021-06-01 11:13:40 +00:00
pool = newClone(AttestationPool.init(dag, quarantine))
state = newClone(dag.headState)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
cache = StateCache()
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
info = ForkedEpochInfo()
refactor batch validation not to require genesis_validators_root each time (#4640)
2023-02-20 08:26:22 +00:00
batchCrypto = BatchCrypto.new(
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
rng, eager = proc(): bool = false,
refactor batch validation not to require genesis_validators_root each time (#4640)
2023-02-20 08:26:22 +00:00
genesis_validators_root = dag.genesis_validators_root, taskpool)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# Slot 0 is a finalized slot - won't be making attestations for it..
check:
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
process_slots(
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
defaultRuntimeConfig, state[], getStateField(state[], slot) + 1,
move `state_transition` to `Result` (#3284) * better error messages in api * avoid `BlockData` copies when replaying blocks
2022-01-17 11:19:58 +00:00
cache, info, {}).isOk()
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
test "Empty committee when no committee for slot":
add sync committee types and some helpers (#2829)
2021-08-28 09:00:00 +00:00
template committee(idx: uint64): untyped =
get_beacon_committee(
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState, dag.head.slot, idx.CommitteeIndex, cache)
add sync committee types and some helpers (#2829)
2021-08-28 09:00:00 +00:00
template committeeLen(idx: uint64): untyped =
get_beacon_committee_len(
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState, dag.head.slot, idx.CommitteeIndex, cache)
add sync committee types and some helpers (#2829)
2021-08-28 09:00:00 +00:00
check:
committee(0).len > 0
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
committee(63).len == 0
add sync committee types and some helpers (#2829)
2021-08-28 09:00:00 +00:00
check:
committeeLen(2) > 0
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
committeeLen(63) == 0
add sync committee types and some helpers (#2829)
2021-08-28 09:00:00 +00:00
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
test "validateAttestation":
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)" This reverts commit 748be8b67ba8598340766cabcce784cc9b0f353e.
2023-05-15 15:41:30 +00:00
var cache: StateCache
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
for blck in makeTestBlocks(
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)" This reverts commit 748be8b67ba8598340766cabcce784cc9b0f353e.
2023-05-15 15:41:30 +00:00
dag.headState, cache, int(SLOTS_PER_EPOCH * 5), attested = false):
Backfill support for ChainDAG (#3171) In the ChainDAG, 3 block pointers are kept: genesis, tail and head. This PR adds one more block pointer: the backfill block which represents the block that has been backfilled so far. When doing a checkpoint sync, a random block is given as starting point - this is the tail block, and we require that the tail block has a corresponding state. When backfilling, we end up with blocks without corresponding states, hence we cannot use `tail` as a backfill pointer - there is no state. Nonetheless, we need to keep track of where we are in the backfill process between restarts, such that we can answer GetBeaconBlocksByRange requests. This PR adds the basic support for backfill handling - it needs to be integrated with backfill sync, and the REST API needs to be adjusted to take advantage of the new backfilled blocks when responding to certain requests. Future work will also enable moving the tail in either direction: * pruning means moving the tail forward in time and removing states * backwards means recreating past states from genesis, such that intermediate states are recreated step by step all the way to the tail - at that point, tail, genesis and backfill will match up. * backfilling is done when backfill != genesis - later, this will be the WSS checkpoint instead
2021-12-13 13:36:06 +00:00
let added = dag.addHeadBlock(verifier, blck.phase0Data) do (
rework spec imports (#2779) The spec imports are a mess to work with, so this branch cleans them up a bit to ensure that we avoid generic sandwitches and that importing stuff generally becomes easier. * reexport crypto/digest/presets because these are part of the public symbol set of the rest of the spec types * don't export `merge` types from `base` - this causes circular deps * fix circular deps in `ssz/spec_types` - this is the first step in disentangling ssz from spec * be explicit about phase0 vs altair - longer term, `altair` will become the "natural" type set, then merge and so on, so no point in giving `phase0` special preferential treatment
2021-08-12 13:08:20 +00:00
blckRef: BlockRef, signedBlock: phase0.TrustedSignedBeaconBlock,
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
epochRef: EpochRef, unrealized: FinalityCheckpoints):
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
# Callback add to fork choice if valid
use BeaconTime instead of Slot in fork choice (#3138) * use v1.1.6 test vectors; use BeaconTime instead of Slot in fork choice * tick through every slot at least once * use div INTERVALS_PER_SLOT and use precomputed constants of them * use correct (even if numerically equal) constant
2021-12-21 18:56:08 +00:00
pool[].addForkChoice(
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
epochRef, blckRef, unrealized, signedBlock.message,
blckRef.slot.start_beacon_time)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
check: added.isOk()
more withdrowls (#4674)
2023-03-02 16:13:35 +00:00
dag.updateHead(added[], quarantine[], [])
singe validator key cache Instead of keeping a validator key list per EpochRef, this PR introduces a single shared validator key list in ChainDAG, and cleans up some other ChainDAG and key-related issues. The PR does not introduce the validator key list in the state transition - this is because we batch-check all signatures before entering the spec code, thus the spec code never hits the cache. A future refactor should _probably_ remove the threadvar altogether. There's a few other small fixes in here that make the flow easier to read: * fix `var ChainDAGRef` -> `ChainDAGRef` * fix `var QuarantineRef` -> `QuarantineRef` * consistent `dag` variable name * avoid using threadvar pubkey cache in most cases * better error messages in batch signature checking
2021-06-01 11:13:40 +00:00
pruneAtFinalization(dag, pool[])
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
var
# Create attestations for slot 1
beacon_committee = get_beacon_committee(
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState, dag.head.slot, 0.CommitteeIndex, cache)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
att_1_0 = makeAttestation(
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState, dag.head.root, beacon_committee[0], cache)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
att_1_1 = makeAttestation(
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
dag.headState, dag.head.root, beacon_committee[1], cache)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
committees_per_slot =
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
get_committee_count_per_slot(
dag.headState, att_1_0.data.slot.epoch, cache)
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
subnet = compute_subnet_for_attestation(
committees_per_slot,
att_1_0.data.slot, att_1_0.data.index.CommitteeIndex)
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
beaconTime = att_1_0.data.slot.start_beacon_time()
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
check:
validateAttestation(pool, batchCrypto, att_1_0, beaconTime, subnet, true).waitFor().isOk
# Same validator again
validateAttestation(pool, batchCrypto, att_1_0, beaconTime, subnet, true).waitFor().error()[0] ==
ValidationResult.Ignore
pool[].nextAttestationEpoch.setLen(0) # reset for test
check:
# Wrong subnet
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
validateAttestation(
pool, batchCrypto, att_1_0, beaconTime, SubnetId(subnet.uint8 + 1), true).waitFor().isErr
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
pool[].nextAttestationEpoch.setLen(0) # reset for test
check:
# Too far in the future
validateAttestation(
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
pool, batchCrypto, att_1_0, beaconTime - 1.seconds, subnet, true).waitFor().isErr
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
pool[].nextAttestationEpoch.setLen(0) # reset for test
check:
# Too far in the past
validateAttestation(
pool, batchCrypto, att_1_0,
beaconTime - (SECONDS_PER_SLOT * SLOTS_PER_EPOCH - 1).int.seconds,
Perform attestation check before broadcast (#2550) Currently, we have a bit of a convoluted flow where when sending attestations, we start broadcasting them over gossip then pass them to the attestation validation to include them in the local attestation pool - it should be the other way around: we should be checking attestations _before_ gossipping them - this serves as an additional safety net to ensure that we don't publish junk - this becomes more important when publishing attestations from the API. Also, the REST API was performing its own validation meaning attestations coming from REST would be validated twice - finally, the JSON RPC wasn't pre-validating and would happily broadcast invalid attestations. * Unified attestation production pipeline with the same flow for gossip, locally and API-produced attestations: all are now validated and entered into the pool, then broadcast/republished * Refactor subnet handling with specific SubnetId alias, streamlining where subnets are computed, avoiding the need to pass around the number of active validators * Move some of the subnet handling code to eth2_network * Use BitArray throughout for subnet handling
2021-05-10 07:13:36 +00:00
subnet, true).waitFor().isErr
remove attestation/aggregate queue (#2519) With the introduction of batching and lazy attestation aggregation, it no longer makes sense to enqueue attestations between the signature check and adding them to the attestation pool - this only takes up valuable CPU without any real benefit. * add successfully validated attestations to attestion pool directly * avoid copying participant list around for single-vote attestations, pass single validator index instead * release decompressed gossip memory earlier, specially during async message validation * use cooked signatures in a few more places to avoid reloads and errors * remove some Defect-raising versions of signature-loading * release decompressed data memory before validating message
2021-04-26 20:39:44 +00:00
block:
var broken = att_1_0
broken.signature.blob[0] += 1
pool[].nextAttestationEpoch.setLen(0) # reset for test
check:
# Invalid signature
validateAttestation(
pool, batchCrypto, broken, beaconTime, subnet, true).waitFor().
error()[0] == ValidationResult.Reject
block:
var broken = att_1_0
broken.signature.blob[5] += 1
pool[].nextAttestationEpoch.setLen(0) # reset for test
# One invalid, one valid (batched)
let
fut_1_0 = validateAttestation(
pool, batchCrypto, broken, beaconTime, subnet, true)
fut_1_1 = validateAttestation(
pool, batchCrypto, att_1_1, beaconTime, subnet, true)
check:
fut_1_0.waitFor().error()[0] == ValidationResult.Reject
fut_1_1.waitFor().isOk()
block:
var broken = att_1_0
# This shouldn't deserialize, which is a different way to break it
broken.signature.blob = default(type broken.signature.blob)
pool[].nextAttestationEpoch.setLen(0) # reset for test
# One invalid, one valid (batched)
let
fut_1_0 = validateAttestation(
pool, batchCrypto, broken, beaconTime, subnet, true)
fut_1_1 = validateAttestation(
pool, batchCrypto, att_1_1, beaconTime, subnet, true)
check:
fut_1_0.waitFor().error()[0] == ValidationResult.Reject
fut_1_1.waitFor().isOk()
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
suite "Gossip validation - Extra": # Not based on preset config
test "validateSyncCommitteeMessage":
const num_validators = SLOTS_PER_EPOCH
harden sync sub committee selection (#2965) * harden sync sub committee selection also turn it into an iterator * fix test and warning
2021-10-07 13:19:47 +00:00
let
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
cfg = block:
var cfg = defaultRuntimeConfig
cfg.ALTAIR_FORK_EPOCH = (GENESIS_EPOCH + 1).Epoch
cfg
batch-verify sync messages for a small perf boost (#3151) * batch-verify sync messages for a small perf boost Generally reuses the same structure as attestation and aggregate verification * normalize `signatures` and `signature_batch` to use the same pattern of verification * normalize parameter names, order etc for signature stuff in general * avoid calling `blsSign` directly - instead, go through `signatures` consistently
2021-12-09 12:56:54 +00:00
taskpool = Taskpool.new()
quarantine = newClone(Quarantine.init())
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
rng = HmacDrbgContext.new()
batch-verify sync messages for a small perf boost (#3151) * batch-verify sync messages for a small perf boost Generally reuses the same structure as attestation and aggregate verification * normalize `signatures` and `signature_batch` to use the same pattern of verification * normalize parameter names, order etc for signature stuff in general * avoid calling `blsSign` directly - instead, go through `signatures` consistently
2021-12-09 12:56:54 +00:00
var
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
verifier = BatchVerifier(rng: rng, taskpool: Taskpool.new())
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
dag = block:
batch-verify sync messages for a small perf boost (#3151) * batch-verify sync messages for a small perf boost Generally reuses the same structure as attestation and aggregate verification * normalize `signatures` and `signature_batch` to use the same pattern of verification * normalize parameter names, order etc for signature stuff in general * avoid calling `blsSign` directly - instead, go through `signatures` consistently
2021-12-09 12:56:54 +00:00
let
Validator monitoring (#2925) Validator monitoring based on and mostly compatible with the implementation in Lighthouse - tracks additional logs and metrics for specified validators so as to stay on top on performance. The implementation works more or less the following way: * Validator pubkeys are singled out for monitoring - these can be running on the node or not * For every action that the validator takes, we record steps in the process such as messages being seen on the network or published in the API * When the dust settles at the end of an epoch, we report the information from one epoch before that, which coincides with the balances being updated - this is a tradeoff between being correct (waiting for finalization) and providing relevant information in a timely manner)
2021-12-20 19:20:31 +00:00
validatorMonitor = newClone(ValidatorMonitor.init())
dag = ChainDAGRef.init(
cfg, makeTestDB(num_validators), validatorMonitor, {})
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
var cache = StateCache()
for blck in makeTestBlocks(
Revert "Revert "accelerate `getShufflingRef` (#4911)" (#4958)" This reverts commit 748be8b67ba8598340766cabcce784cc9b0f353e.
2023-05-15 15:41:30 +00:00
dag.headState, cache, int(SLOTS_PER_EPOCH),
attested = false, cfg = cfg):
harden sync sub committee selection (#2965) * harden sync sub committee selection also turn it into an iterator * fix test and warning
2021-10-07 13:19:47 +00:00
let added =
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
case blck.kind
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560) * `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` * revert unrelated change * revert unrelated changes * update test summaries
2023-01-28 19:53:41 +00:00
of ConsensusFork.Phase0:
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
const nilCallback = OnPhase0BlockAdded(nil)
Backfill support for ChainDAG (#3171) In the ChainDAG, 3 block pointers are kept: genesis, tail and head. This PR adds one more block pointer: the backfill block which represents the block that has been backfilled so far. When doing a checkpoint sync, a random block is given as starting point - this is the tail block, and we require that the tail block has a corresponding state. When backfilling, we end up with blocks without corresponding states, hence we cannot use `tail` as a backfill pointer - there is no state. Nonetheless, we need to keep track of where we are in the backfill process between restarts, such that we can answer GetBeaconBlocksByRange requests. This PR adds the basic support for backfill handling - it needs to be integrated with backfill sync, and the REST API needs to be adjusted to take advantage of the new backfilled blocks when responding to certain requests. Future work will also enable moving the tail in either direction: * pruning means moving the tail forward in time and removing states * backwards means recreating past states from genesis, such that intermediate states are recreated step by step all the way to the tail - at that point, tail, genesis and backfill will match up. * backfilling is done when backfill != genesis - later, this will be the WSS checkpoint instead
2021-12-13 13:36:06 +00:00
dag.addHeadBlock(verifier, blck.phase0Data, nilCallback)
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560) * `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` * revert unrelated change * revert unrelated changes * update test summaries
2023-01-28 19:53:41 +00:00
of ConsensusFork.Altair:
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
const nilCallback = OnAltairBlockAdded(nil)
Backfill support for ChainDAG (#3171) In the ChainDAG, 3 block pointers are kept: genesis, tail and head. This PR adds one more block pointer: the backfill block which represents the block that has been backfilled so far. When doing a checkpoint sync, a random block is given as starting point - this is the tail block, and we require that the tail block has a corresponding state. When backfilling, we end up with blocks without corresponding states, hence we cannot use `tail` as a backfill pointer - there is no state. Nonetheless, we need to keep track of where we are in the backfill process between restarts, such that we can answer GetBeaconBlocksByRange requests. This PR adds the basic support for backfill handling - it needs to be integrated with backfill sync, and the REST API needs to be adjusted to take advantage of the new backfilled blocks when responding to certain requests. Future work will also enable moving the tail in either direction: * pruning means moving the tail forward in time and removing states * backwards means recreating past states from genesis, such that intermediate states are recreated step by step all the way to the tail - at that point, tail, genesis and backfill will match up. * backfilling is done when backfill != genesis - later, this will be the WSS checkpoint instead
2021-12-13 13:36:06 +00:00
dag.addHeadBlock(verifier, blck.altairData, nilCallback)
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560) * `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` * revert unrelated change * revert unrelated changes * update test summaries
2023-01-28 19:53:41 +00:00
of ConsensusFork.Bellatrix:
rename On{Merge,Bellatrix}BlockAdded and Rollback{Merge,Bellatrix}HashedProc (#3321)
2022-01-26 12:21:29 +00:00
const nilCallback = OnBellatrixBlockAdded(nil)
rename mergeData to bellatrixData and mergeFork to bellatrixFork (#3315)
2022-01-24 16:23:13 +00:00
dag.addHeadBlock(verifier, blck.bellatrixData, nilCallback)
`BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` (#4560) * `BeaconStateFork/BeaconBlockFork` -> `ConsensusFork` * revert unrelated change * revert unrelated changes * update test summaries
2023-01-28 19:53:41 +00:00
of ConsensusFork.Capella:
capella gossip support (#4386)
2022-12-04 07:42:03 +00:00
const nilCallback = OnCapellaBlockAdded(nil)
dag.addHeadBlock(verifier, blck.capellaData, nilCallback)
rename ConsensusFork.EIP4844 to ConsensusFork.Deneb (#4692)
2023-03-04 13:35:39 +00:00
of ConsensusFork.Deneb:
rename `OnEIP4844BlockAdded` > `OnDenebBlockAdded` (#4714)
2023-03-10 17:14:27 +00:00
const nilCallback = OnDenebBlockAdded(nil)
rename most eip4844Data to denebData (#4693)
2023-03-04 22:23:52 +00:00
dag.addHeadBlock(verifier, blck.denebData, nilCallback)
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
check: added.isOk()
more withdrowls (#4674)
2023-03-02 16:13:35 +00:00
dag.updateHead(added[], quarantine[], [])
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
dag
refactor batch validation not to require genesis_validators_root each time (#4640)
2023-02-20 08:26:22 +00:00
let batchCrypto = BatchCrypto.new(
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
rng, eager = proc(): bool = false,
refactor batch validation not to require genesis_validators_root each time (#4640)
2023-02-20 08:26:22 +00:00
genesis_validators_root = dag.genesis_validators_root, taskpool)
var
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
state = assignClone(dag.headState.altairData)
Bugfix: Avoid the aggregation of duplicate signatures when creating sync committee contributions
2021-11-04 18:22:24 +00:00
slot = state[].data.slot
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
Better REST/RPC error messages (#3046) * Better REST/RPC error messages * homogenise block logging (root first) * homegenise message verification pipeline (verify in `gossip_verification`, act in `eth2_processor`) * use `subcommitteeIdx` consistently * log each sent contribution * fix block_sim * fix block topic * don't recalc root on gossip block validation * move position loop into sync pool
2021-11-05 15:39:47 +00:00
subcommitteeIdx = 0.SyncSubcommitteeIndex
Bugfix: Avoid the aggregation of duplicate signatures when creating sync committee contributions
2021-11-04 18:22:24 +00:00
syncCommittee = @(dag.syncCommitteeParticipants(slot))
Better REST/RPC error messages (#3046) * Better REST/RPC error messages * homogenise block logging (root first) * homegenise message verification pipeline (verify in `gossip_verification`, act in `eth2_processor`) * use `subcommitteeIdx` consistently * log each sent contribution * fix block_sim * fix block topic * don't recalc root on gossip block validation * move position loop into sync pool
2021-11-05 15:39:47 +00:00
subcommittee = toSeq(syncCommittee.syncSubcommittee(subcommitteeIdx))
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
index = subcommittee[0]
expectedCount = subcommittee.count(index)
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
pubkey = state[].data.validators.item(index).pubkey
Implementation of the Keymanager API (BETA) https://github.com/ethereum/keymanager-APIs
2021-12-22 12:37:31 +00:00
keystoreData = KeystoreData(kind: KeystoreKind.Local,
load suggested fee recipient file also when keymanager is disabled (#4078) Since these files may have been created in a previous run or manually, we want to keep loading them even on nodes that don't enable the keystore API (for example static setups) Other changes: * log keystore loading progressively (#3699) * print initial fee recipient when loading validators * log dynamic fee recipient updates
2022-09-17 05:30:07 +00:00
pubkey: pubkey,
Implementation of the Keymanager API (BETA) https://github.com/ethereum/keymanager-APIs
2021-12-22 12:37:31 +00:00
privateKey: MockPrivKeys[index])
load suggested fee recipient file also when keymanager is disabled (#4078) Since these files may have been created in a previous run or manually, we want to keep loading them even on nodes that don't enable the keystore API (for example static setups) Other changes: * log keystore loading progressively (#3699) * print initial fee recipient when loading validators * log dynamic fee recipient updates
2022-09-17 05:30:07 +00:00
validator = AttachedValidator(
Fix a potential segfault and various potential stalls (#4003) * Fixes a segfault during block production when the Keymanager API is disabled. The Keymanager is now disabled on half of the local testnet nodes to catch such problems in the future. * Fixes multiple potential stalls from REST requests being done without a timeout. From practice, we know that such requests can hang forever if not cancelled with a timeout. At best, this would be a resource leak, at worst, it may lead to a full stall of the client and missed validator duties. * Changes some Options usages to Opt (for easier use of valueOr)
2022-08-19 21:51:30 +00:00
kind: ValidatorKind.Local, data: keystoreData, index: Opt.some index)
cleanups (#3819) * avoid circular panda imports * move deposit merkleization helpers to spec/ * normalize validator signature helpers to spec names / params * remove redundant functions for remote signing
2022-06-29 16:53:59 +00:00
resMsg = waitFor getSyncCommitteeMessage(
use correct root in `test_gossip_validation` (#4951) Use block root not state root for sync committee message in test.
2023-05-14 14:18:50 +00:00
validator, state[].data.fork, state[].data.genesis_validators_root,
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
slot, state[].latest_block_root)
Remote signing client/server. (#3077)
2021-11-30 01:20:21 +00:00
msg = resMsg.get()
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
reduce `nim-eth` dependencies just for RNG (#5099) We have several modules that import `nim-eth` for the sole purpose of its `keys.newRng` function. This function is meanwhile a simple wrapper around `nim-bearssl`'s `HmacDrbgContext.new()`, so the import doesn't really serve a use anymore. Replace `keys.newRng` with the direct call to reduce `nim-eth` imports.
2023-06-19 22:43:50 +00:00
syncCommitteePool = newClone(SyncCommitteeMsgPool.init(rng, cfg))
batch-verify sync messages for a small perf boost (#3151) * batch-verify sync messages for a small perf boost Generally reuses the same structure as attestation and aggregate verification * normalize `signatures` and `signature_batch` to use the same pattern of verification * normalize parameter names, order etc for signature stuff in general * avoid calling `blsSign` directly - instead, go through `signatures` consistently
2021-12-09 12:56:54 +00:00
res = waitFor validateSyncCommitteeMessage(
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
dag, quarantine, batchCrypto, syncCommitteePool,
msg, subcommitteeIdx, slot.start_beacon_time(), true)
(bid, cookedSig, positions) = res.get()
Better REST/RPC error messages (#3046) * Better REST/RPC error messages * homogenise block logging (root first) * homegenise message verification pipeline (verify in `gossip_verification`, act in `eth2_processor`) * use `subcommitteeIdx` consistently * log each sent contribution * fix block_sim * fix block topic * don't recalc root on gossip block validation * move position loop into sync pool
2021-11-05 15:39:47 +00:00
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
syncCommitteePool[].addSyncCommitteeMessage(
Better REST/RPC error messages (#3046) * Better REST/RPC error messages * homogenise block logging (root first) * homegenise message verification pipeline (verify in `gossip_verification`, act in `eth2_processor`) * use `subcommitteeIdx` consistently * log each sent contribution * fix block_sim * fix block topic * don't recalc root on gossip block validation * move position loop into sync pool
2021-11-05 15:39:47 +00:00
msg.slot,
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
bid,
Better REST/RPC error messages (#3046) * Better REST/RPC error messages * homogenise block logging (root first) * homegenise message verification pipeline (verify in `gossip_verification`, act in `eth2_processor`) * use `subcommitteeIdx` consistently * log each sent contribution * fix block_sim * fix block topic * don't recalc root on gossip block validation * move position loop into sync pool
2021-11-05 15:39:47 +00:00
msg.validator_index,
cookedSig,
subcommitteeIdx,
positions)
let
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
contrib = block:
let contrib = (ref SignedContributionAndProof)()
Implement a missing ingnore rule for sync committee contributions (#3941)
2022-08-09 09:52:11 +00:00
check:
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
syncCommitteePool[].produceContribution(
slot, bid, subcommitteeIdx,
contrib.message.contribution)
syncCommitteePool[].addContribution(
contrib[], bid,
contrib.message.contribution.signature.load.get)
cleanups (#3819) * avoid circular panda imports * move deposit merkleization helpers to spec/ * normalize validator signature helpers to spec names / params * remove redundant functions for remote signing
2022-06-29 16:53:59 +00:00
let signRes = waitFor validator.getContributionAndProofSignature(
state[].data.fork, state[].data.genesis_validators_root,
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
contrib[].message)
Remote signing client/server. (#3077)
2021-11-30 01:20:21 +00:00
doAssert(signRes.isOk())
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
contrib[].signature = signRes.get()
contrib
use correct slot when producing sync aggregate around forks (#5089) `produceSyncAggregate` is called in new slot when block is produced, while the other functions in `sync_committee_msg_pool` are called in previous slot. So, need to subtract 1 slot when producing sync aggregate to accept the signatures using the old digest during fork transition.
2023-06-16 21:30:36 +00:00
aggregate = syncCommitteePool[].produceSyncAggregate(bid, slot + 1)
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
check:
expectedCount > 1 # Cover edge case
res.isOk
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
contrib.message.contribution.aggregation_bits.countOnes == expectedCount
handle duplicate pubkeys in sync committee (#2902) When sync committee message handling was introduced in #2830, the edge case of the same validator being selected multiple times as part of a sync subcommittee was not covered. Not handling that edge case makes sync contributions have a lower-than-expected participation rate as each sync validator is only counted up through once per subcommittee. This patch ensures that this edge case is properly covered.
2021-09-28 07:44:20 +00:00
aggregate.sync_committee_bits.countOnes == expectedCount
Better REST/RPC error messages (#3046) * Better REST/RPC error messages * homogenise block logging (root first) * homegenise message verification pipeline (verify in `gossip_verification`, act in `eth2_processor`) * use `subcommitteeIdx` consistently * log each sent contribution * fix block_sim * fix block topic * don't recalc root on gossip block validation * move position loop into sync pool
2021-11-05 15:39:47 +00:00
# Same message twice should be ignored
validateSyncCommitteeMessage(
segregate sync committee messages by period / fork (#4953) `SyncCommitteeMsgPool` grouped messages by their `beacon_block_root`. This is problematic around sync committee period boundaries and forks. Around sync committee period boundaries, members from both the current and next sync committee may sign the same `beacon_block_root`; mixing the signatures from both committees together is a mistake. Likewise, around fork transitions, the `signing_root` changes, so those messages also need to be segregated.
2023-05-17 04:55:55 +00:00
dag, quarantine, batchCrypto, syncCommitteePool,
msg, subcommitteeIdx, state[].data.slot.start_beacon_time(), true
).waitFor().isErr()