status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/spec/beaconstate.nim

1166 lines
48 KiB
Nim
Raw Normal View History

move spec stuff to spec directory helps maintain sanity in the face of spec changes
2018-11-28 19:49:03 +00:00
# beacon_chain
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
# Copyright (c) 2018-2022 Status Research & Development GmbH
move spec stuff to spec directory helps maintain sanity in the face of spec changes
2018-11-28 19:49:03 +00:00
# Licensed and distributed under either of
replace links to Apache and MIT licenses with HTTPS versions (#592)
2019-11-25 15:30:02 +00:00
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
move spec stuff to spec directory helps maintain sanity in the face of spec changes
2018-11-28 19:49:03 +00:00
# at your option. This file may not be copied, modified, or distributed except according to those terms.
compatibility with Nim 1.4+ (#3888)
2022-07-29 10:53:42 +00:00
when (NimMajor, NimMinor) < (1, 4):
{.push raises: [Defect].}
else:
{.push raises: [].}
spec: regulate exceptions (#913) * spec: regulate exceptions * a few more simple raises
2020-04-22 05:53:02 +00:00
move spec stuff to spec directory helps maintain sanity in the face of spec changes
2018-11-28 19:49:03 +00:00
import
Cache and resend, rather than recreate, builder API registrations (#4040)
2022-08-31 00:29:03 +00:00
std/[algorithm, collections/heapqueue, math, sequtils, tables],
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
stew/assign2,
fix broken metrics during replay (#2090) * move metrics out of state transition * add validator count metric * remove expensive beacon_current_validators, beacon_previous_validators metrics (they should be reimplemented with cache), add cheap beacon_active_validators to approximate * remove unused validator count metrics * tidy imports/defects
2020-11-27 22:16:13 +00:00
json_serialization/std/sets,
chronicles,
disentangle eth2 types from the ssz library (#2785) * reorganize ssz dependencies This PR continues the work in https://github.com/status-im/nimbus-eth2/pull/2646, https://github.com/status-im/nimbus-eth2/pull/2779 as well as past issues with serialization and type, to disentangle SSZ from eth2 and at the same time simplify imports and exports with a structured approach. The principal idea here is that when a library wants to introduce SSZ support, they do so via 3 files: * `ssz_codecs` which imports and reexports `codecs` - this covers the basic byte conversions and ensures no overloads get lost * `xxx_merkleization` imports and exports `merkleization` to specialize and get access to `hash_tree_root` and friends * `xxx_ssz_serialization` imports and exports `ssz_serialization` to specialize ssz for a specific library Those that need to interact with SSZ always import the `xxx_` versions of the modules and never `ssz` itself so as to keep imports simple and safe. This is similar to how the REST / JSON-RPC serializers are structured in that someone wanting to serialize spec types to REST-JSON will import `eth2_rest_serialization` and nothing else. * split up ssz into a core library that is independendent of eth2 types * rename `bytes_reader` to `codec` to highlight that it contains coding and decoding of bytes and native ssz types * remove tricky List init overload that causes compile issues * get rid of top-level ssz import * reenable merkleization tests * move some "standard" json serializers to spec * remove `ValidatorIndex` serialization for now * remove test_ssz_merkleization * add tests for over/underlong byte sequences * fix broken seq[byte] test - seq[byte] is not an SSZ type There are a few things this PR doesn't solve: * like #2646 this PR is weak on how to handle root and other dontSerialize fields that "sometimes" should be computed - the same problem appears in REST / JSON-RPC etc * Fix a build problem on macOS * Another way to fix the macOS builds Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 18:57:58 +00:00
../extras,
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
./datatypes/[phase0, altair, bellatrix],
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
"."/[eth2_merkleization, forks, signatures, validator]
move spec stuff to spec directory helps maintain sanity in the face of spec changes
2018-11-28 19:49:03 +00:00
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
from ./datatypes/capella import BeaconState, ExecutionPayloadHeader, Withdrawal
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
export extras, forks, validator
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#increase_balance
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
func increase_balance*(balance: var Gwei, delta: Gwei) =
balance += delta
0.6.1 - signing_root, deposits, validator indices and committee count (#261) * signed_root -> signing_root * implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput * update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more * re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
2019-05-09 12:27:37 +00:00
func increase_balance*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: var ForkyBeaconState, index: ValidatorIndex, delta: Gwei) =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Increase the validator balance at index ``index`` by ``delta``.
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if delta != 0: # avoid dirtying the balance cache if not needed
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
increase_balance(state.balances.mitem(index), delta)
spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator (#159) * spec version 0.4.0 update: shard_block_root -> crosslink_data_root; switch from validator.slashed_epoch -> validator.slashed approach; a couple assert -> doAssert; note fix for underflow-prone spec changes in checkAttestation; GENESIS_SLOT change from 2^63 to 2^32; refactor Proposer and ProposalSlashing structure; add signed_root; remove duplicate slashValidator * re-apply shard_block_root -> crosslink_data_root * remove incorrect humaneSlotNum * add (run-time only, alas) sanity check on signed_root
2019-03-08 17:44:31 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#decrease_balance
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
func decrease_balance*(balance: var Gwei, delta: Gwei) =
balance =
if delta > balance:
0'u64
else:
balance - delta
0.6.1 - signing_root, deposits, validator indices and committee count (#261) * signed_root -> signing_root * implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput * update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more * re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
2019-05-09 12:27:37 +00:00
func decrease_balance*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: var ForkyBeaconState, index: ValidatorIndex, delta: Gwei) =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Decrease the validator balance at index ``index`` by ``delta``, with
## underflow protection.
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if delta != 0: # avoid dirtying the balance cache if not needed
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
decrease_balance(state.balances.mitem(index), delta)
0.6.1 - signing_root, deposits, validator indices and committee count (#261) * signed_root -> signing_root * implement new process_deposit; update timing parameters to 0.6.1; update Deposit to 0.6.1 and remove DepositInput * update get_active_validator_indices and get_epoch_committee_count to 0.6.1; rm get_current_epoch_committee_count, get_previous_epoch_committee_count, and get_next_epoch_committee_count; bump state_sim default validator count a bit more * re-introduce 0.5.1-ish get_active_validator_indices, get_epoch_committee_count as scaffolding for still-0.5.1ish shuffling
2019-05-09 12:27:37 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#deposits
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#modified-process_deposit
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
func get_validator_from_deposit*(deposit: DepositData):
v0.12.2 beacon chain protocol update (#1378)
2020-07-27 10:59:57 +00:00
Validator =
let
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
amount = deposit.amount
v0.12.2 beacon chain protocol update (#1378)
2020-07-27 10:59:57 +00:00
effective_balance = min(
amount - amount mod EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE)
Validator(
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
pubkey: deposit.pubkey,
withdrawal_credentials: deposit.withdrawal_credentials,
v0.12.2 beacon chain protocol update (#1378)
2020-07-27 10:59:57 +00:00
activation_eligibility_epoch: FAR_FUTURE_EPOCH,
activation_epoch: FAR_FUTURE_EPOCH,
exit_epoch: FAR_FUTURE_EPOCH,
withdrawable_epoch: FAR_FUTURE_EPOCH,
effective_balance: effective_balance
)
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#compute_activation_exit_epoch
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
func compute_activation_exit_epoch*(epoch: Epoch): Epoch =
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
## Return the epoch during which validator activations and exits initiated in
## ``epoch`` take effect.
initial 0.9.0 spec sync (#509) * rename compute_epoch_of_slot(...) to compute_epoch_at_slot(...) * remove some unnecessary imports; remove some crosslink-related code and tests; complete renaming of compute_epoch_of_slot(...) to compute_epoch_at_slot(...) * rm more transfer-related code and tests; rm more unnecessary strutils imports * rm remaining unused imports * remove useless get_empty_per_epoch_cache(...)/compute_start_slot_of_epoch(...) calls * rename compute_start_slot_of_epoch(...) to compute_start_slot_at_epoch(...) * rename ACTIVATION_EXIT_DELAY to MAX_SEED_LOOKAHEAD * update domain types to 0.9.0 * mark AttesterSlashing, IndexedAttestation, AttestationDataAndCustodyBit, DepositData, BeaconBlockHeader, Fork, integer_squareroot(...), and process_voluntary_exit(...) as 0.9.0 * mark increase_balance(...), decrease_balance(...), get_block_root(...), CheckPoint, Deposit, PendingAttestation, HistoricalBatch, is_active_validator(...), and is_slashable_attestation_data(...) as 0.9.0 * mark compute_activation_exit_epoch(...), bls_verify(...), Validator, get_active_validator_indices(...), get_current_epoch(...), get_total_active_balance(...), and get_previous_epoch(...) as 0.9.0 * mark get_block_root_at_slot(...), ProposerSlashing, get_domain(...), VoluntaryExit, mainnet preset Gwei values, minimal preset max operations, process_block_header(...), and is_slashable_validator(...) as 0.9.0 * mark makeWithdrawalCredentials(...), get_validator_churn_limit(...), get_total_balance(...), is_valid_indexed_attestation(...), bls_aggregate_pubkeys(...), initial genesis value/constants, Attestation, get_randao_mix(...), mainnet preset max operations per block constants, minimal preset Gwei values and time parameters, process_eth1_data(...), get_shuffled_seq(...), compute_committee(...), and process_slots(...) as 0.9.0; partially update get_indexed_attestation(...) to 0.9.0 by removing crosslink refs and associated tests * mark initiate_validator_exit(...), process_registry_updates(...), BeaconBlock, Eth1Data, compute_domain(...), process_randao(...), process_attester_slashing(...), get_base_reward(...), and process_slot(...) as 0.9.0
2019-10-30 19:41:19 +00:00
epoch + 1 + MAX_SEED_LOOKAHEAD
epochs replace slots as main first-class entity; several mechanical renamings (ForkData -> Fork; .fork_data -> .fork; remove stub custody types/code; some type aliasing to keep slots and epochs more enforceably separate; fix test_state_transition.nim assumption that GENESIS_SLOT == 0
2019-01-29 03:22:22 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#get_validator_churn_limit
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
func get_validator_churn_limit*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
cfg: RuntimeConfig, state: ForkyBeaconState, cache: var StateCache):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
uint64 =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return the validator churn limit for the current epoch.
cache beacon committee size calculation (#1363) * cache beacon committee size calculation this fixes a bug in get_validator_churn_limit as well * fix * make committee counts consistently uint64 mixing feels like the worst of the two worlds
2020-07-23 17:01:07 +00:00
max(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg.MIN_PER_EPOCH_CHURN_LIMIT,
cache beacon committee size calculation (#1363) * cache beacon committee size calculation this fixes a bug in get_validator_churn_limit as well * fix * make committee counts consistently uint64 mixing feels like the worst of the two worlds
2020-07-23 17:01:07 +00:00
count_active_validators(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
state, state.get_current_epoch(), cache) div cfg.CHURN_LIMIT_QUOTIENT)
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#initiate_validator_exit
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
func initiate_validator_exit*(
cfg: RuntimeConfig, state: var ForkyBeaconState,
index: ValidatorIndex, cache: var StateCache): Result[void, cstring] =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Initiate the exit of the validator with index ``index``.
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
if state.validators.item(index).exit_epoch != FAR_FUTURE_EPOCH:
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
return ok() # Before touching cache
REST/JSON-RPC: speed up several requests (#3092) REST/JSON-RPC and a few more also invalidate caches unnecessarily, similar to https://github.com/status-im/nimbus-eth2/pull/3089 * avoid copying validator on balance request
2021-11-12 22:29:28 +00:00
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
# Return if validator already initiated exit
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let validator = addr state.validators.mitem(index)
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
tone down validator exit logging (#1939) Validators exiting is normal, no need to scream about it * avoid reallocating seq on big exit queue * avoid fetching state cache when updating head (it's rarely needed) * remove incorrectly implemented live validator counts (avoids memory allocs)
2020-11-02 17:34:23 +00:00
trace "Validator exiting",
index = index,
num_validators = state.validators.len,
current_epoch = get_current_epoch(state),
validator_slashed = validator.slashed,
validator_withdrawable_epoch = validator.withdrawable_epoch,
validator_exit_epoch = validator.exit_epoch,
validator_effective_balance = validator.effective_balance
var exit_queue_epoch = compute_activation_exit_epoch(get_current_epoch(state))
# Compute max exit epoch
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
for idx in 0..<state.validators.len:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let exit_epoch = state.validators.item(idx).exit_epoch
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
if exit_epoch != FAR_FUTURE_EPOCH and exit_epoch > exit_queue_epoch:
exit_queue_epoch = exit_epoch
tone down validator exit logging (#1939) Validators exiting is normal, no need to scream about it * avoid reallocating seq on big exit queue * avoid fetching state cache when updating head (it's rarely needed) * remove incorrectly implemented live validator counts (avoids memory allocs)
2020-11-02 17:34:23 +00:00
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
var
exit_queue_churn: int
for idx in 0..<state.validators.len:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
if state.validators.item(idx).exit_epoch == exit_queue_epoch:
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
exit_queue_churn += 1
tone down validator exit logging (#1939) Validators exiting is normal, no need to scream about it * avoid reallocating seq on big exit queue * avoid fetching state cache when updating head (it's rarely needed) * remove incorrectly implemented live validator counts (avoids memory allocs)
2020-11-02 17:34:23 +00:00
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
if exit_queue_churn.uint64 >= get_validator_churn_limit(cfg, state, cache):
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
exit_queue_epoch += 1
# Set validator exit epoch and withdrawable epoch
validator.exit_epoch = exit_queue_epoch
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
if validator.exit_epoch + cfg.MIN_VALIDATOR_WITHDRAWABILITY_DELAY <
validator.exit_epoch:
return err("initiate_validator_exit: exit_epoch overflowed")
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
validator.withdrawable_epoch =
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
validator.exit_epoch + cfg.MIN_VALIDATOR_WITHDRAWABILITY_DELAY
spec updates (#37) * use repeat_hash from spec in randao * add some logging to state processing * export crypto string converters in spec insulation layer * enable block signature verification * ssz: add support for arrays, remove custom data type code * correctly handle previous and new block * add trivial block processing tests * prefer iterative repeat_hash * state transition mostly done * handle most specials and slashings
2018-12-13 16:00:55 +00:00
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
ok()
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#modified-slash_validator
CL spec URL updates to v1.2.0-rc.1 (#3655)
2022-05-23 19:30:24 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0-rc.1/specs/bellatrix/beacon-chain.md#modified-slash_validator
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func get_slashing_penalty*(state: ForkyBeaconState,
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
validator_effective_balance: Gwei): Gwei =
# TODO Consider whether this is better than splitting the functions apart; in
# each case, tradeoffs. Here, it's just changing a couple of constants.
when state is phase0.BeaconState:
validator_effective_balance div MIN_SLASHING_PENALTY_QUOTIENT
elif state is altair.BeaconState:
validator_effective_balance div MIN_SLASHING_PENALTY_QUOTIENT_ALTAIR
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
elif state is bellatrix.BeaconState:
rename *_{MERGE => BELLATRIX} constant names (#3296)
2022-01-18 16:31:05 +00:00
validator_effective_balance div MIN_SLASHING_PENALTY_QUOTIENT_BELLATRIX
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
elif state is capella.BeaconState:
validator_effective_balance div MIN_SLASHING_PENALTY_QUOTIENT_BELLATRIX
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
else:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
{.fatal: "invalid BeaconState type".}
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#modified-slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/bellatrix/beacon-chain.md#modified-slash_validator
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
func get_whistleblower_reward*(validator_effective_balance: Gwei): Gwei =
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
validator_effective_balance div WHISTLEBLOWER_REWARD_QUOTIENT
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#modified-slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/bellatrix/beacon-chain.md#modified-slash_validator
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
func get_proposer_reward(state: ForkyBeaconState, whistleblower_reward: Gwei): Gwei =
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
when state is phase0.BeaconState:
whistleblower_reward div PROPOSER_REWARD_QUOTIENT
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
elif state is altair.BeaconState or state is bellatrix.BeaconState or
state is capella.BeaconState:
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
whistleblower_reward * PROPOSER_WEIGHT div WEIGHT_DENOMINATOR
else:
More efficient reward data persistance; Address review comments The new format is based on compressed CSV files in two channels: * Detailed per-epoch data * Aggregated "daily" summaries The use of append-only CSV file speeds up significantly the epoch processing speed during data generation. The use of compression results in smaller storage requirements overall. The use of the aggregated files has a very minor cost in both CPU and storage, but leads to near interactive speed for report generation. Other changes: - Implemented support for graceful shut downs to avoid corrupting the saved files. - Fixed a memory leak caused by lacking `StateCache` clean up on each iteration. - Addressed review comments - Moved the rewards and penalties calculation code in a separate module Required invasive changes to existing modules: - The `data` field of the `KeyedBlockRef` type is made public to be used by the validator rewards monitor's Chain DAG update procedure. - The `getForkedBlock` procedure from the `blockchain_dag.nim` module is made public to be used by the validator rewards monitor's Chain DAG update procedure.
2022-01-17 12:58:33 +00:00
{.fatal: "invalid BeaconState type".}
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#modified-slash_validator
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/bellatrix/beacon-chain.md#modified-slash_validator
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
proc slash_validator*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
cfg: RuntimeConfig, state: var ForkyBeaconState,
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
slashed_index: ValidatorIndex, cache: var StateCache):
Result[void, cstring] =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Slash the validator with index ``index``.
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
let epoch = get_current_epoch(state)
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
? initiate_validator_exit(cfg, state, slashed_index, cache)
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
let validator = addr state.validators.mitem(slashed_index)
add validator_leaving logging at debug chronicles level for all call paths to initiate_validator_exit(...)
2019-11-25 08:22:16 +00:00
tone down validator exit logging (#1939) Validators exiting is normal, no need to scream about it * avoid reallocating seq on big exit queue * avoid fetching state cache when updating head (it's rarely needed) * remove incorrectly implemented live validator counts (avoids memory allocs)
2020-11-02 17:34:23 +00:00
trace "slash_validator: ejecting validator via slashing (validator_leaving)",
add validator_leaving logging at debug chronicles level for all call paths to initiate_validator_exit(...)
2019-11-25 08:22:16 +00:00
index = slashed_index,
num_validators = state.validators.len,
current_epoch = get_current_epoch(state),
validator_slashed = validator.slashed,
validator_withdrawable_epoch = validator.withdrawable_epoch,
validator_exit_epoch = validator.exit_epoch,
validator_effective_balance = validator.effective_balance
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
validator.slashed = true
validator.withdrawable_epoch =
max(validator.withdrawable_epoch, epoch + EPOCHS_PER_SLASHINGS_VECTOR)
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
state.slashings.mitem(int(epoch mod EPOCHS_PER_SLASHINGS_VECTOR)) +=
More 0.8.0 updates (#311) * replace BeaconState.finalized_{epoch,root} with BeaconState.finalized_checkpoint; rename get_delayed_activation_exit_epoch(...) to compute_activation_exit_epoch(...) and mark as 0.8.0; update get_churn_limit(...)/get_validator_churn_limit(...) to 0.8.0; update process_registry_updates(...) to 0.8.0 * update process_crosslinks(...) to 0.8.0; mark compute_start_slot_of_epoch(...) and get_committee_count(...) as 0.8.0 * mark Fork, is_slashable_validator(...), and get_beacon_proposer_index(...) as 0.8.0 * rename LATEST_SLASHED_EXIT_LENGTH to EPOCHS_PER_SLASHINGS_VECTOR; update process_slashings(...) to 0.8.0; remove pointless type conversion warning in get_previous_epoch(...) * convert remaining references to finalized_epoch to finalized_checkpoint.epoch * update slash_validator(...) to 0.8.0; mark inital value, Gwei, and time constants as 0.8.0; mark hash(...) and processBlockHeader(...) as 0.8.0 * rename WHISTLEBLOWING_REWARD_QUOTIENT to WHISTLEBLOWER_REWARD_QUOTIENT; rename LATEST_ACTIVE_INDEX_ROOTS_LENGTH to EPOCHS_PER_HISTORICAL_VECTOR (randao will also get merged into this); remove get_active_index_root(...); mark time parameter, signature domain types, and max operations per block constants as 0.8.0; update rewards and penalties constants to 0.8.0 * update is_valid_indexed_attestation(...) to 0.8.0; mark process_slot(...) as 0.8.0 * replace BeaconState.{current,previous}_justified_{epoch,root} with BeaconState.{current,previous}_justified_checkpoint
2019-07-05 08:30:05 +00:00
validator.effective_balance
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
decrease_balance(state, slashed_index,
get_slashing_penalty(state, validator.effective_balance))
More spec v0.3.0 notations and updates (#129) * tighten get_active_index_root epoch assertion bound * implement merkle_root * rm process_penalties_and_exits; add slash_validator * rm rm'd and unused constant
2019-02-20 20:35:27 +00:00
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
# The rest doesn't make sense without there being any proposer index, so skip
use shuffled validator cache in more places; cleanups (#1095)
2020-06-03 05:42:08 +00:00
let proposer_index = get_beacon_proposer_index(state, cache)
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
if proposer_index.isNone:
debug "No beacon proposer index and probably no active validators"
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
return ok()
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
halve memory usage from state caches
2020-07-08 08:56:25 +00:00
# Apply proposer and whistleblower rewards
epochs replace slots as main first-class entity; several mechanical renamings (ForkData -> Fork; .fork_data -> .fork; remove stub custody types/code; some type aliasing to keep slots and epochs more enforceably separate; fix test_state_transition.nim assumption that GENESIS_SLOT == 0
2019-01-29 03:22:22 +00:00
let
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
# Spec has whistleblower_index as optional param, but it's never used.
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
whistleblower_index = proposer_index.get
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
whistleblower_reward = get_whistleblower_reward(validator.effective_balance)
proposer_reward = get_proposer_reward(state, whistleblower_reward)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
increase_balance(state, proposer_index.get, proposer_reward)
Attester slashing - split process_attester_slashing/processAttesterSlashings + tests (#424) * attester slashing tests - pending #415 * split process_attester_slashing/processAttesterSlashings * Add logs to attester_slashing * deactivate bls tests for now (https://github.com/status-im/nim-beacon-chain/issues/429) and cherry-pick from https://github.com/status-im/nim-beacon-chain/pull/403/commits/60f2437dd26a758df56baafb8c513639fe838e82
2019-09-10 22:03:06 +00:00
# TODO: evaluate if spec bug / underflow can be triggered
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
doAssert(whistleblower_reward >= proposer_reward, "Spec bug: underflow in slash_validator")
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
increase_balance(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
state, whistleblower_index, whistleblower_reward - proposer_reward)
initial process_penalties_and_exits
2019-01-16 13:39:34 +00:00
consensus specs v1.2.0-rc.1 (#3661)
2022-05-25 13:49:29 +00:00
ok()
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
func genesis_time_from_eth1_timestamp*(cfg: RuntimeConfig, eth1_timestamp: uint64): uint64 =
eth1_timestamp + cfg.GENESIS_DELAY
Switch to monitoring of Blocks; More accurate genesis detection and faster Eth1 syncing
2020-06-27 12:01:19 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#genesis
rename initialize_beacon_state to initialize_beacon_state_from_eth1 (#2536)
2021-05-04 10:19:11 +00:00
proc initialize_beacon_state_from_eth1*(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig,
more interop fixes * fix eth1 interop block hash * update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 10:31:14 +00:00
eth1_block_hash: Eth2Digest,
eth1_timestamp: uint64,
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
deposits: openArray[DepositData],
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
flags: UpdateFlags = {}): phase0.BeaconState =
convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return' (#132) * convert some asserts to doAsserts to keep them in release mode builds; rename get_initial_beacon_state to get_genesis_beacon_state to track spec; switch target spec version to 0.3.0; switch references to penalize_validator to slash_validator/slashValidator to track spec; make some function returns safer by omitting 'return' * 2x shuffling speedup by hoisting pivot calculations per https://github.com/protolambda/eth2-shuffle
2019-02-22 09:56:45 +00:00
## Get the genesis ``BeaconState``.
beacon state updates * add `on_startup` genesis state constructor * update state transition to use ssz * update validator adder
2018-11-29 05:23:40 +00:00
##
## Before the beacon chain starts, validators will register in the Eth1 chain
## and deposit ETH. When enough many validators have registered, a
## `ChainStart` log will be emitted and the beacon chain can start beaconing.
##
## Because the state root hash is part of the genesis block, the beacon state
## must be calculated before creating the genesis block.
spec updates (#37) * use repeat_hash from spec in randao * add some logging to state processing * export crypto string converters in spec insulation layer * enable block signature verification * ssz: add support for arrays, remove custom data type code * correctly handle previous and new block * add trivial block processing tests * prefer iterative repeat_hash * state transition mostly done * handle most specials and slashings
2018-12-13 16:00:55 +00:00
beacon state updates * add `on_startup` genesis state constructor * update state transition to use ssz * update validator adder
2018-11-29 05:23:40 +00:00
# Induct validators
spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch (#127) * spec 0.3.0 updates: EPOCH_LENGTH -> SLOTS_PER_EPOCH, SlotNumber -> Slot, EpochNumber -> Epoch
2019-02-20 01:33:58 +00:00
# Not in spec: the system doesn't work unless there are at least SLOTS_PER_EPOCH
spec updates * more work on block processing * fix some unsigned issues * fix ssz issue when last chunk is not full * work around empty const seq compiler bug * XXX -> TODO
2018-11-29 22:11:05 +00:00
# validators - there needs to be at least one member in each committee -
# good to know for testing, though arguably the system is not that useful at
# at that point :)
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) * avoid converting from uint64 to int, and where most feasible, int type conversion at all * .len.uint64 -> .len64 * fix 32-bit compilation * try keeping state_sim loop variable/bounds as int for 32-bit Azure * len64 -> lenu64
2020-07-26 18:55:48 +00:00
doAssert deposits.lenu64 >= SLOTS_PER_EPOCH
beacon state updates * add `on_startup` genesis state constructor * update state transition to use ssz * update validator adder
2018-11-29 05:23:40 +00:00
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
# TODO https://github.com/nim-lang/Nim/issues/19094
template state(): untyped = result
state = phase0.BeaconState(
forkedbeaconstate_helpers -> forks (#2772) Simpler module name for stuff that covers forks * check that runtime config matches database state * also include some assorted altair cleanups * use "standard" genesis fork in local testnet to work around missing runtime config support
2021-08-10 20:46:35 +00:00
fork: genesisFork(cfg),
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
genesis_time: genesis_time_from_eth1_timestamp(cfg, eth1_timestamp),
more interop fixes * fix eth1 interop block hash * update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 10:31:14 +00:00
eth1_data:
Eth1Data(block_hash: eth1_block_hash, deposit_count: uint64(len(deposits))),
0.8.0, continued (#315) * rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0 * mark get_shard_delta(...) as 0.8.0 * mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0 * move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 12:23:02 +00:00
latest_block_header:
Fix genesis blockheader zero signature (supercedes #395) (#400)
2019-09-08 00:56:24 +00:00
BeaconBlockHeader(
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
body_root: hash_tree_root(default(phase0.BeaconBlockBody))))
beacon state updates * add `on_startup` genesis state constructor * update state transition to use ssz * update validator adder
2018-11-29 05:23:40 +00:00
re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1 (#569) * re-enable test_interop based on zcli with 0.9.1 specs and update initialize_beacon_state_from_eth1(...) to 0.9.1 * switch many procs to funcs * fix import os.nim instead; ospaths is deprecated [Deprecated] warnings
2019-11-21 09:15:10 +00:00
# Seed RANDAO with Eth1 entropy
state.randao_mixes.fill(eth1_block_hash)
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
var merkleizer = createMerkleizer(2'i64^DEPOSIT_CONTRACT_TREE_DEPTH)
for i, deposit in deposits:
let htr = hash_tree_root(deposit)
merkleizer.addChunk(htr.data)
# This is already known in the Eth1 monitor, but it would be too
# much work to refactor all the existing call sites in the test suite
state.eth1_data.deposit_root = mixInLength(merkleizer.getFinalHash(),
deposits.len)
state.eth1_deposit_index = deposits.lenu64
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
var pubkeyToIndex = initTable[ValidatorPubKey, ValidatorIndex]()
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
for idx, deposit in deposits:
let
pubkey = deposit.pubkey
amount = deposit.amount
pubkeyToIndex.withValue(pubkey, foundIdx) do:
# Increase balance by deposit amount
fix type-conversions-to-self, unexport from nimbus_beacon_node, and rm unused vars/procs (#3211)
2021-12-20 11:21:17 +00:00
increase_balance(state, foundIdx[], amount)
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
do:
if skipBlsValidation in flags or
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
verify_deposit_signature(cfg, deposit):
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
pubkeyToIndex[pubkey] = ValidatorIndex(state.validators.len)
Reset cached indices when resetting cache on SSZ read (#2480) * Reset cached indices when resetting cache on SSZ read When deserializing into an existing structure, the cache should be cleared - goes for json also. Also improve error messages.
2021-04-08 10:11:04 +00:00
if not state.validators.add(get_validator_from_deposit(deposit)):
raiseAssert "too many validators"
if not state.balances.add(amount):
raiseAssert "same as validators"
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
else:
# Invalid deposits are perfectly possible
trace "Skipping deposit with invalid signature",
deposit = shortLog(deposit)
stop routing through update_validator_status
2019-01-16 12:20:44 +00:00
0.8.0, continued (#315) * rename get_genesis_beacon_state(...) to initialize_beacon_state_from_eth1(...); remove unused vestiges get_temporary_block_header(...) and get_empty_block(...); partly align initialize_beacon_state_from_eth1(...) with 0.8.0 version; implement CompactCommittee object type; update process_final_updates(...) to 0.8.0 * mark get_shard_delta(...) as 0.8.0 * mark get_total_active_balance(...) and epoch transition helper functions as 0.8.0 * move FORK_CHOICE_BALANCE_INCREMENT, not in spec anymore, to sole user in fork_choice
2019-07-10 12:23:02 +00:00
# Process activations
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
for vidx in state.validators.vindices:
more interop fixes * fix eth1 interop block hash * update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 10:31:14 +00:00
let
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
balance = state.balances.item(vidx)
validator = addr state.validators.mitem(vidx)
more interop fixes * fix eth1 interop block hash * update initialize_beacon_state_from_eth1 to 0.8.3
2019-09-02 10:31:14 +00:00
validator.effective_balance = min(
balance - balance mod EFFECTIVE_BALANCE_INCREMENT, MAX_EFFECTIVE_BALANCE)
if validator.effective_balance == MAX_EFFECTIVE_BALANCE:
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
validator.activation_eligibility_epoch = GENESIS_EPOCH
validator.activation_epoch = GENESIS_EPOCH
spec updates (#37) * use repeat_hash from spec in randao * add some logging to state processing * export crypto string converters in spec insulation layer * enable block signature verification * ssz: add support for arrays, remove custom data type code * correctly handle previous and new block * add trivial block processing tests * prefer iterative repeat_hash * state transition mostly done * handle most specials and slashings
2018-12-13 16:00:55 +00:00
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
# Set genesis validators root for domain separation and chain versioning
Progress towards #991; Enable the distinct ssz.List type
2020-05-18 17:49:22 +00:00
state.genesis_validators_root = hash_tree_root(state.validators)
initial 0.11.0 spec version update
2020-03-14 21:54:45 +00:00
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
# TODO https://github.com/nim-lang/Nim/issues/19094
# state
spec updates (#37) * use repeat_hash from spec in randao * add some logging to state processing * export crypto string converters in spec insulation layer * enable block signature verification * ssz: add support for arrays, remove custom data type code * correctly handle previous and new block * add trivial block processing tests * prefer iterative repeat_hash * state transition mostly done * handle most specials and slashings
2018-12-13 16:00:55 +00:00
begin removing plain BeaconState versions of state transition functions (#951) * remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions * convert tests/test_state_transition.nim to use HashedBeaconState * convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 06:44:19 +00:00
proc initialize_hashed_beacon_state_from_eth1*(
Implement split preset/config support (#2710) * Implement split preset/config support This is the initial bulk refactor to introduce runtime config values in a number of places, somewhat replacing the existing mechanism of loading network metadata. It still needs more work, this is the initial refactor that introduces runtime configuration in some of the places that need it. The PR changes the way presets and constants work, to match the spec. In particular, a "preset" now refers to the compile-time configuration while a "cfg" or "RuntimeConfig" is the dynamic part. A single binary can support either mainnet or minimal, but not both. Support for other presets has been removed completely (can be readded, in case there's need). There's a number of outstanding tasks: * `SECONDS_PER_SLOT` still needs fixing * loading custom runtime configs needs redoing * checking constants against YAML file * yeerongpilly support `build/nimbus_beacon_node --network=yeerongpilly --discv5:no --log-level=DEBUG` * load fork epoch from config * fix fork digest sent in status * nicer error string for request failures * fix tools * one more * fixup * fixup * fixup * use "standard" network definition folder in local testnet Files are loaded from their standard locations, including genesis etc, to conform to the format used in the `eth2-networks` repo. * fix launch scripts, allow unknown config values * fix base config of rest test * cleanups * bundle mainnet config using common loader * fix spec links and names * only include supported preset in binary * drop yeerongpilly, add altair-devnet-0, support boot_enr.yaml
2021-07-12 13:01:38 +00:00
cfg: RuntimeConfig,
begin removing plain BeaconState versions of state transition functions (#951) * remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions * convert tests/test_state_transition.nim to use HashedBeaconState * convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 06:44:19 +00:00
eth1_block_hash: Eth2Digest,
eth1_timestamp: uint64,
Use O(n) algorithm in initialize_beacon_state_from_eth1; Avoid unnecessary merkle proofs generation
2020-11-14 21:43:27 +00:00
deposits: openArray[DepositData],
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
flags: UpdateFlags = {}): phase0.HashedBeaconState =
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
# TODO https://github.com/nim-lang/Nim/issues/19094
result = phase0.HashedBeaconState(
data: initialize_beacon_state_from_eth1(
cfg, eth1_block_hash, eth1_timestamp, deposits, flags))
result.root = hash_tree_root(result.data)
begin removing plain BeaconState versions of state transition functions (#951) * remove near-duplicate code paths: process_slot(), process_slots(), and state_transition() for BeaconState are now wrappers around the HashedBeaconState versions * convert tests/test_state_transition.nim to use HashedBeaconState * convert mocking infrastructure and spec_block/epoch_processing tests to use HashedBeaconBlock, and remove thus unused process_slot*(state: var BeaconState)
2020-04-30 06:44:19 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#genesis-block
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
func get_initial_beacon_block*(state: phase0.HashedBeaconState):
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase0.TrustedSignedBeaconBlock =
Db1 cont (#2573) * Revert "Revert "Upgrade database schema" (#2570)" This reverts commit 6057c2ffb4a481d3994c8fe5a46e7b754d12ae8f. * ssz: fix loading empty lists into existing instances Not a problem earlier because we didn't reuse instances * bump nim-eth * bump nim-web3
2021-05-17 16:37:26 +00:00
# The genesis block is implicitly trusted
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let message = phase0.TrustedBeaconBlock(
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
slot: state.data.slot,
state_root: state.root)
Add support for starting from weak subjectivity checkpoints Also removes the `genesis.ssz` file stored in the data folder. The `medalla-fast-sync` target has been adapted to use the new features.
2020-09-22 20:42:42 +00:00
# parent_root, randao_reveal, eth1_data, signature, and body automatically
# initialized to default values.
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
phase0.TrustedSignedBeaconBlock(
message: message, root: hash_tree_root(message))
fix BlsValue inits
2019-09-08 18:27:09 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#initialize-state-for-pure-altair-testnets-and-test-vectors
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
func get_initial_beacon_block*(state: altair.HashedBeaconState):
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
altair.TrustedSignedBeaconBlock =
# The genesis block is implicitly trusted
let message = altair.TrustedBeaconBlock(
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
slot: state.data.slot,
state_root: state.root)
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
# parent_root, randao_reveal, eth1_data, signature, and body automatically
# initialized to default values.
altair.TrustedSignedBeaconBlock(
message: message, root: hash_tree_root(message))
spec URL updates (#3254)
2022-01-06 18:35:38 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.7/specs/merge/beacon-chain.md#testing
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
func get_initial_beacon_block*(state: bellatrix.HashedBeaconState):
bellatrix.TrustedSignedBeaconBlock =
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
# The genesis block is implicitly trusted
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
let message = bellatrix.TrustedBeaconBlock(
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
slot: state.data.slot,
Restore the build support of the -d:has_genesis_detection feature
2021-12-23 14:58:54 +00:00
state_root: state.root)
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
# parent_root, randao_reveal, eth1_data, signature, and body automatically
# initialized to default values.
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
bellatrix.TrustedSignedBeaconBlock(
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
message: message, root: hash_tree_root(message))
add Capella support to Forked* (#4276) * add Capella support to Forked* * remove cruft * add `OnForkyBlockAdded`
2022-11-02 16:23:30 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.7/specs/merge/beacon-chain.md#testing
func get_initial_beacon_block*(state: capella.HashedBeaconState):
capella.TrustedSignedBeaconBlock =
# The genesis block is implicitly trusted
let message = capella.TrustedBeaconBlock(
slot: state.data.slot,
state_root: state.root)
# parent_root, randao_reveal, eth1_data, signature, and body automatically
# initialized to default values.
capella.TrustedSignedBeaconBlock(
message: message, root: hash_tree_root(message))
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
func get_initial_beacon_block*(state: ForkedHashedBeaconState):
ForkedTrustedSignedBeaconBlock =
withState(state):
allow accessing withState forky state via `forkyState` (#4026)
2022-08-26 14:14:18 +00:00
ForkedTrustedSignedBeaconBlock.init(get_initial_beacon_block(forkyState))
Support starting from altair (#3054) * Support starting from altair * hide `finalized-checkpoint-` - they are incomplete and usage may cause crashes * remove genesis detection code (broken, obsolete) * enable starting ChainDAG from altair checkpoints - this is a prerequisite for checkpoint sync (TODO: backfill) * tighten checkpoint state conditions * show error when starting from checkpoint with existing database (not supported) * print rest-compatible JSON in ncli/state_sim * altair/merge support in ncli * more altair/merge support in ncli_db * pre-load header to speed up loading * fix forked block decoding
2021-11-10 11:39:08 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#get_block_root_at_slot
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
func get_block_root_at_slot*(state: ForkyBeaconState, slot: Slot): Eth2Digest =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return the block root at a recent ``slot``.
more v0.2.0 spec updates (#107) * switch to v0.2.0 shuffling scheme * fix some >80 char lines; add get_total_balance helper function (used in epoch processing, etc); note intended removal of repeat_hash and users thereof; add some int_to_bytes and bytes_to_int helpers; more underflow comments; add Gwei type; adjust generate_seed to include additional entropy source; note egregious inefficiencies * minor code cleanup * move some general helper functions to helpers.nim and clarify little-endian comments
2019-02-13 10:26:32 +00:00
address issue #1580 (#1600) * address issue #1580 * Update beacon_chain/spec/beaconstate.nim Co-authored-by: Jacek Sieka <jacek@status.im>
2020-09-02 16:16:25 +00:00
# Potential overflow/wrap shouldn't occur, as get_block_root_at_slot() called
# from internally controlled sources, but flag this explicitly, in case.
doAssert slot + SLOTS_PER_HISTORICAL_ROOT > slot
begin 0.5.0 spec update (#179) * begin 0.5.0 spec update: parent_root -> previous_block_root, BeaconState.justified_epoch -> BeaconState.current_justified_epoch, DOMAIN_PROPOSAL -> DOMAIN_BEACON_BLOCK, temporarily rename BeaconBlockHeader to BeaconBlockHeaderRLP to allow for gradual re-merging without disrupting RLP; mark a few unchanged functions and data types, implement get_temporary_block_header/get_empty_block/should_update_validator_registry/processBlockHeader/cacheState; update a few others * a dozen or so more trivial mostly comment changes, finding more unchanged parts of 0.5.0 * several more trivial changes; goal is to reduce noise around the upcoming substantial changes (epoch processing, etc)
2019-03-16 19:52:37 +00:00
doAssert state.slot <= slot + SLOTS_PER_HISTORICAL_ROOT
spec updates (#39) * some minor renames * fix some leftover block slot references (replaced by state slot) * fix some more uint underflows * update epoch calculations (reward et al) * now works past 100 epochs at least with empty blocks (very slow)
2018-12-14 16:12:39 +00:00
doAssert slot < state.slot
Initial frozen phase 0/v0.8.0 spec (#305) * rename slot_to_epoch(...) to compute_epoch_of_slot(...) * rename aggregation_bitfield/custody_bitfield fields to aggregation_bits/custody_bits; rename convert_to_indexed(...) to get_indexed_attestation(...); mark BeaconBlockHeader as 0.8.0; update minimal preset MIN_ATTESTATION_INCLUSION_DELAY/time parameters in general to 0.8 * fix beacon node compilation; it used slightly different capitalizations of slot_to_epoch/slotToEpoch * mark integer_squareroot(...), Deposit, VoluntaryExit, and Transfer as 0.8.0; rename get_epoch_start_slot(...) to compute_start_slot_of_epoch(...) * add new Domain alias for uint64; rename bls_domain(...) to compute_domain(...), MAX_INDICES_PER_ATTESTATION to MAX_VALIDATORS_PER_COMMITTEE, and SignatureDomain to DomainType; update get_domain(...) to 0.8.0 * update/mark initiate_validator_exit(...) and Crosslink as 0.8.0; rename BeaconState.latest_block_roots -> BeaconState.block_roots; mark HistoricalBatch as 0.8.0
2019-07-01 07:53:42 +00:00
state.block_roots[slot mod SLOTS_PER_HISTORICAL_ROOT]
beacon state updates * add `on_startup` genesis state constructor * update state transition to use ssz * update validator adder
2018-11-29 05:23:40 +00:00
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
func get_block_root_at_slot*(
state: ForkedHashedBeaconState, slot: Slot): Eth2Digest =
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
## Return the block root at a recent ``slot``.
withState(state):
allow accessing withState forky state via `forkyState` (#4026)
2022-08-26 14:14:18 +00:00
get_block_root_at_slot(forkyState.data, slot)
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#get_block_root
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
func get_block_root*(state: ForkyBeaconState, epoch: Epoch): Eth2Digest =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Return the block root at the start of a recent ``epoch``.
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
get_block_root_at_slot(state, epoch.start_slot())
More 0.6.1 updates (#274) * implement get_churn_limit from 0.6.1; update initiate_validator_exit and get_previous_epoch to 0.6.1; remove obsoleted/non-spec reduce_balance, replaced by decrease_balance; mark BeaconBlock as 0.6.1 * rename get_block_root to get_block_root_at_slot and introduce epoch-oriented get_block_root; implement get_attestation_slot, get_matching_source_attestations, get_matching_target_attestations, get_matching_head_attestations, 0.6.1 get_attesting_indices, get_unslashed_attesting_indices, get_attestation_deltas, process_rewards_and_penalties; rm get_inactivity_penalty * update Validator and processVoluntaryExits to 0.6.1; rm obsolete inclusion_slots/inclusion_distances * rm removed activate_validator; mark DepositData, misc values, Gwei values, Randao processing, state caching as 0.6.1; update GENESIS_SLOT to 64, since it doesn't quite yet work at 0 * mark BeaconBlockHeader as 0.6.1; update BeaconBlockBody to 0.6.1; rename WHISTLEBLOWER_REWARD_QUOTIENT to WHISTLEBLOWING_REWARD_QUOTIENT; update slash_validator to 0.6.1 * implement 0.6.2 is_slashable_validator; update processProposerSlashings to 0.6.2; mark state caching as 0.6.2 * mark get_total_active_balance and process_slashings as 0.6.2; update get_base_reward to 0.6.2 * rm prepare_validator_for_withdrawal, process_exit_queue; mark mainnet misc constants as 0.6.2; update process block header to 0.6.2 * address mratsim's code review comment
2019-05-29 10:08:56 +00:00
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
func get_block_root*(state: ForkedHashedBeaconState, epoch: Epoch): Eth2Digest =
## Return the block root at the start of a recent ``epoch``.
withState(state):
allow accessing withState forky state via `forkyState` (#4026)
2022-08-26 14:14:18 +00:00
get_block_root(forkyState.data, epoch)
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#get_total_balance
40% faster Altair epoch slot processing (#2814) * construct all unslashed, participating balances in one validator scan * remove altair benchmarking setup for block_sim * revert a benchmarking change from proc to func * remove more benchmarking func/proc tweaks * re-add asSeq, which is necessary because unsafeAddr * ... except for the block_sim benchmark part
2021-08-25 14:43:00 +00:00
template get_total_balance(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: ForkyBeaconState, validator_indices: untyped): Gwei =
0.11.1 beacon chain spec update (#836) * initial 0.11.1 spec commit; no test regressions and finalizes in eth2_network_simulation * with BLS 0.10/0.11 available, stop skipping attester slashing, proposer slashing, and voluntary exist operations fixture tests * switch param orders to group state.{fork, genesis_validators_root}; bump spec/datatypes spec version for network purposes * mark attestation construction and broadcast and some minimal/mainnet constants as 0.11.1-compatible; remove phase 1 sharding constants from minimal which don't exist in that preset
2020-03-30 11:31:44 +00:00
## Return the combined effective balance of the ``indices``.
## ``EFFECTIVE_BALANCE_INCREMENT`` Gwei minimum to avoid divisions by zero.
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
## Math safe up to ~10B ETH, after which this overflows uint64.
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
var res = 0.Gwei
for validator_index in validator_indices:
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
res += state.validators[validator_index].effective_balance
double altair epoch slot processing speed (#2811) * approximately double altair epoch processing speed * don't redundantly clear state balances cache * only invalidate altair state balances cache once in process_rewards_and_penalties()
2021-08-24 20:09:03 +00:00
max(EFFECTIVE_BALANCE_INCREMENT, res)
switch from legacy pre-spec sum_effective_balances(...) to get_total_balance(...)
2019-02-22 18:46:45 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#is_eligible_for_activation_queue
test fixture cleanups (#2976) * "official" -> "scenarios", like the submodule * fewer test binaries - various compile hacks have been improved over time, test suite should follow * remove obsolete bls tests - there are better test vectors in nim-blscurve * remove obsolete mentions of `ssz_testing` * remove obsolete comments about proc vs globals, unittest2 already uses proc's
2021-10-12 11:36:52 +00:00
func is_eligible_for_activation_queue*(validator: Validator): bool =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Check if ``validator`` is eligible to be placed into the activation queue.
initial 0.9.3 spec update
2019-12-16 18:08:50 +00:00
validator.activation_eligibility_epoch == FAR_FUTURE_EPOCH and
validator.effective_balance == MAX_EFFECTIVE_BALANCE
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#is_eligible_for_activation
group justified and finalized `Checkpoint` (#3841) The justified and finalized `Checkpoint` are frequently passed around together. This introduces a new `FinalityCheckpoint` data structure that combines them into one. Due to the large usage of this structure in fork choice, also took this opportunity to update fork choice tests to the latest v1.2.0-rc.1 spec. Many additional tests enabled, some need more work, e.g. EL mock blocks. Also implemented `discard_equivocations` which was skipped in #3661, and improved code reuse across fork choice logic while at it.
2022-07-06 10:33:02 +00:00
func is_eligible_for_activation*(
state: ForkyBeaconState, validator: Validator): bool =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Check if ``validator`` is eligible for activation.
initial 0.9.3 spec update
2019-12-16 18:08:50 +00:00
# Placement in queue is finalized
validator.activation_eligibility_epoch <= state.finalized_checkpoint.epoch and
# Has not yet been activated
validator.activation_epoch == FAR_FUTURE_EPOCH
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#is_valid_indexed_attestation
lazily initialize validator public keys, which were a third or more of sync CPU usage
2020-07-23 14:38:28 +00:00
proc is_valid_indexed_attestation*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: ForkyBeaconState, indexed_attestation: SomeIndexedAttestation,
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
flags: UpdateFlags): Result[void, cstring] =
mark comments documenting functions as a whole as such (#1613)
2020-09-08 08:54:55 +00:00
## Check if ``indexed_attestation`` is not empty, has sorted and unique
## indices and has a valid aggregate signature.
fix is_valid_indexed_attestation() to check for too-high attestation indices
2020-06-22 10:37:47 +00:00
template is_sorted_and_unique(s: untyped): bool =
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
var res = true
fix is_valid_indexed_attestation() to check for too-high attestation indices
2020-06-22 10:37:47 +00:00
for i in 1 ..< s.len:
if s[i - 1].uint64 >= s[i].uint64:
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
res = false
break
res
fix is_valid_indexed_attestation() to check for too-high attestation indices
2020-06-22 10:37:47 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
if len(indexed_attestation.attesting_indices) == 0:
return err("indexed_attestation: no attesting indices")
fix is_valid_indexed_attestation() to check for too-high attestation indices
2020-06-22 10:37:47 +00:00
# Not from spec, but this function gets used in front-line roles, not just
# behind firewall.
Prefer converting int` to uint64 and switch foo.len.uint64 to .len64 (#1375) * avoid converting from uint64 to int, and where most feasible, int type conversion at all * .len.uint64 -> .len64 * fix 32-bit compilation * try keeping state_sim loop variable/bounds as int for 32-bit Azure * len64 -> lenu64
2020-07-26 18:55:48 +00:00
let num_validators = state.validators.lenu64
fix is_valid_indexed_attestation() to check for too-high attestation indices
2020-06-22 10:37:47 +00:00
if anyIt(indexed_attestation.attesting_indices, it >= num_validators):
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
return err("indexed attestation: not all indices valid validators")
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
if not is_sorted_and_unique(indexed_attestation.attesting_indices):
return err("indexed attestation: indices not sorted and unique")
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
More safe 0.7.0 updates (#283) * complete marking unchanged-in-0.7.0 beaconstate functions * complete marking unchanged-in-0.7.0 spec refs functions in helpers.nim and time.nim: get_randao_mix(...), bytes_to_int(...), and description of beacon chain processing apropos time * complete remaining feasible 0.7.0 updates which lack externally visible effect
2019-06-14 07:50:14 +00:00
# Verify aggregate signature
stylecheck fixes (#3593)
2022-04-14 15:39:37 +00:00
if not (skipBlsValidation in flags or indexed_attestation.signature is TrustedSig):
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
let pubkeys = mapIt(
indexed_attestation.attesting_indices, state.validators[it].pubkey)
collect signature production and verificaiton in one place (#1179) * collect signature production and verificaiton in one place Signatures are made over data and domain - here we collect all such activities in one place. Also: * security: fix cast-before-range-check * log block/attestation verification consistently * run block verification based on `getProposer` in its own history * clean up some unused stuff * import * missing raises
2020-06-16 05:45:04 +00:00
if not verify_attestation_signature(
state.fork, state.genesis_validators_root, indexed_attestation.data,
pubkeys, indexed_attestation.signature):
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
return err("indexed attestation: signature verification failure")
simplify validator api (#573) * avoid passing whole state to validator api (easier when it goes out-of-process) * clean up some cruft
2019-11-21 09:57:59 +00:00
small spec cleanups (#1501) * clean up logging a bit * return error on indexed attestation check
2020-08-13 13:47:06 +00:00
ok()
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#get_attesting_indices
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
func get_attesting_indices*(state: ForkyBeaconState,
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
data: AttestationData,
bits: CommitteeValidatorsBits,
cache: var StateCache): seq[ValidatorIndex] =
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
## Return the set of attesting indices corresponding to ``data`` and ``bits``
## or nothing if `data` is invalid
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
var res: seq[ValidatorIndex]
# Can't be an iterator due to https://github.com/nim-lang/Nim/issues/18188
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
let committee_index = CommitteeIndex.init(data.index)
if committee_index.isErr() or bits.lenu64 != get_beacon_committee_len(
state, data.slot, committee_index.get(), cache):
trace "get_attesting_indices: invalid attestation data"
misc memory and perf fixes (#1899) * misc memory and perf fixes * use EpochRef for attestation aggregation * compress effective balances in memory (medalla unfinalized: 4gb -> 1gb) * avoid hitting db when rewinding to head or clearance state * avoid hitting db when blocks can be applied to in-memory state - speeds up startup considerably * avoid storing epochref in fork choice * simplify and speed up beacon block creation flow - avoids state reload thanks to head rewind optimization * iterator-based committee and attestation participation help avoid lots of small memory allocations throughout epoch transition (40% speedup on epoch processing, for example during startup) * add constant for threshold
2020-10-22 10:53:33 +00:00
else:
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
for index_in_committee, validator_index in get_beacon_committee(
Implement all sync committee duties in the validator client (#3583) Other changes: * logtrace can now verify sync committee messages and contributions * Many unnecessary use of pairs() have been removed for consistency * Map 40x BN response codes to BeaconNodeStatus.Incompatible in the VC
2022-05-10 10:03:40 +00:00
state, data.slot, committee_index.get(), cache):
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
if bits[index_in_committee]:
res.add validator_index
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
res
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
func get_attesting_indices*(state: ForkedHashedBeaconState;
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
data: AttestationData;
bits: CommitteeValidatorsBits;
cache: var StateCache): seq[ValidatorIndex] =
# TODO when https://github.com/nim-lang/Nim/issues/18188 fixed, use an
# iterator
var idxBuf: seq[ValidatorIndex]
withState(state):
allow accessing withState forky state via `forkyState` (#4026)
2022-08-26 14:14:18 +00:00
for vidx in forkyState.data.get_attesting_indices(data, bits, cache):
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
idxBuf.add vidx
idxBuf
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
proc is_valid_indexed_attestation*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: ForkyBeaconState, attestation: SomeAttestation, flags: UpdateFlags,
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
cache: var StateCache): Result[void, cstring] =
# This is a variation on `is_valid_indexed_attestation` that works directly
# with an attestation instead of first constructing an `IndexedAttestation`
# and then validating it - for the purpose of validating the signature, the
# order doesn't matter and we can proceed straight to validating the
# signature instead
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
let sigs = attestation.aggregation_bits.countOnes()
if sigs == 0:
return err("is_valid_indexed_attestation: no attesting indices")
# Verify aggregate signature
stylecheck fixes (#3593)
2022-04-14 15:39:37 +00:00
if not (skipBlsValidation in flags or attestation.signature is TrustedSig):
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
var
pubkeys = newSeqOfCap[ValidatorPubKey](sigs)
for index in get_attesting_indices(
state, attestation.data, attestation.aggregation_bits, cache):
pubkeys.add(state.validators[index].pubkey)
if not verify_attestation_signature(
state.fork, state.genesis_validators_root, attestation.data,
pubkeys, attestation.signature):
return err("indexed attestation: signature verification failure")
ok()
Trusted blocks (#1227) * cleanups * fix ncli state root check flag * add block dump to ncli_db * limit ncli_db benchmark length * tone down finalization logs * introduce trusted blocks We only store blocks whose signature we've verified in the database - as such, there's no need to check it again, and most importantly, no need to deserialize the signature when loading from database. 50x startup time improvement, 200x block load time improvement. * fix rewinding when deposits have invalid signature * speed up ancestor iteration by avoiding copy * avoid deserializing signatures for trusted data * load blocks lazily when rewinding (less memory used) * chronicles workarounds * document trustedbeaconblock
2020-06-25 10:23:10 +00:00
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
# Attestation validation
# ------------------------------------------------------------------------------------------
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#attestations
update spec ref URLs (#3979)
2022-08-17 11:33:19 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0-rc.2/specs/phase0/p2p-interface.md#beacon_attestation_subnet_id
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
func check_attestation_slot_target*(data: AttestationData): Result[Slot, cstring] =
time: spring cleaning (#3262) Time in the beacon chain is expressed relative to the genesis time - this PR creates a `beacon_time` module that collects helpers and utilities for dealing the time units - the new module does not deal with actual wall time (that's remains in `beacon_clock`). Collecting the time related stuff in one place makes it easier to find, avoids some circular imports and allows more easily identifying the code actually needs wall time to operate. * move genesis-time-related functionality into `spec/beacon_time` * avoid using `chronos.Duration` for time differences - it does not support negative values (such as when something happens earlier than it should) * saturate conversions between `FAR_FUTURE_XXX`, so as to avoid overflows * fix delay reporting in validator client so it uses the expected deadline of the slot, not "closest wall slot" * simplify looping over the slots of an epoch * `compute_start_slot_at_epoch` -> `start_slot` * `compute_epoch_at_slot` -> `epoch` A follow-up PR will (likely) introduce saturating arithmetic for the time units - this is merely code moves, renames and fixing of small bugs.
2022-01-11 10:01:54 +00:00
if not (data.target.epoch == epoch(data.slot)):
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
return err("Target epoch doesn't match attestation slot")
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
ok(data.slot)
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884)
2020-10-20 08:54:11 +00:00
func check_attestation_target_epoch(
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
data: AttestationData, current_epoch: Epoch): Result[Epoch, cstring] =
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
if not (data.target.epoch == get_previous_epoch(current_epoch) or
data.target.epoch == current_epoch):
return err("Target epoch not current or previous epoch")
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
ok(data.target.epoch)
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
func check_attestation_inclusion(attestation_slot: Slot,
update to 1.0.0-rc.0 message ID; unexport funcs; enable test in mainnet; remove unused subscribe-with-nondummy-handler function; fix gossip received metric (#1884)
2020-10-20 08:54:11 +00:00
current_slot: Slot): Result[void, cstring] =
address issue #1580 (#1600) * address issue #1580 * Update beacon_chain/spec/beaconstate.nim Co-authored-by: Jacek Sieka <jacek@status.im>
2020-09-02 16:16:25 +00:00
# Check for overflow
static:
doAssert SLOTS_PER_EPOCH >= MIN_ATTESTATION_INCLUSION_DELAY
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
if attestation_slot + SLOTS_PER_EPOCH <= attestation_slot:
address issue #1580 (#1600) * address issue #1580 * Update beacon_chain/spec/beaconstate.nim Co-authored-by: Jacek Sieka <jacek@status.im>
2020-09-02 16:16:25 +00:00
return err("attestation data.slot overflow, malicious?")
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
if not (attestation_slot + MIN_ATTESTATION_INCLUSION_DELAY <= current_slot):
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
return err("Attestation too new")
[Split fork choice PR] Derisk-ed attestation checks changes (#1154) * Derisked attestation pool improvements * tune down frequent logs * VoteTracker logging
2020-06-10 06:58:12 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
if not (current_slot <= attestation_slot + SLOTS_PER_EPOCH):
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
return err("Attestation too old")
ok()
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
func check_attestation_index*(
index, committees_per_slot: uint64):
Result[CommitteeIndex, cstring] =
CommitteeIndex.init(index, committees_per_slot)
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
func check_attestation_index*(
data: AttestationData, committees_per_slot: uint64):
Result[CommitteeIndex, cstring] =
check_attestation_index(data.index, committees_per_slot)
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#get_attestation_participation_flag_indices
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
func get_attestation_participation_flag_indices(
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState,
data: AttestationData, inclusion_delay: uint64): seq[int] =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the flag indices that are satisfied by an attestation.
let justified_checkpoint =
if data.target.epoch == get_current_epoch(state):
state.current_justified_checkpoint
else:
state.previous_justified_checkpoint
# Matching roots
let
is_matching_source = data.source == justified_checkpoint
is_matching_target = is_matching_source and data.target.root == get_block_root(state, data.target.epoch)
is_matching_head = is_matching_target and data.beacon_block_root == get_block_root_at_slot(state, data.slot)
# TODO probably this needs to be robustly failable
doAssert is_matching_source
var participation_flag_indices: seq[int]
if is_matching_source and inclusion_delay <= integer_squareroot(SLOTS_PER_EPOCH):
participation_flag_indices.add(TIMELY_SOURCE_FLAG_INDEX)
if is_matching_target and inclusion_delay <= SLOTS_PER_EPOCH:
participation_flag_indices.add(TIMELY_TARGET_FLAG_INDEX)
if is_matching_head and inclusion_delay == MIN_ATTESTATION_INCLUSION_DELAY:
participation_flag_indices.add(TIMELY_HEAD_FLAG_INDEX)
participation_flag_indices
# TODO these aren't great here
# TODO these duplicate some stuff in state_transition_epoch which uses TotalBalances
# better to centralize around that if feasible
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#get_total_active_balance
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
func get_total_active_balance*(state: ForkyBeaconState, cache: var StateCache): Gwei =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the combined effective balance of the active validators.
# Note: ``get_total_balance`` returns ``EFFECTIVE_BALANCE_INCREMENT`` Gwei
# minimum to avoid divisions by zero.
let epoch = state.get_current_epoch()
get_total_balance(
state, cache.get_shuffled_active_validator_indices(state, epoch))
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#get_base_reward_per_increment
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
func get_base_reward_per_increment_sqrt*(
total_active_balance_sqrt: uint64): Gwei =
EFFECTIVE_BALANCE_INCREMENT * BASE_REWARD_FACTOR div total_active_balance_sqrt
update some spec refs to v1.1.0-alpha.8 (#2689)
2021-07-01 08:55:16 +00:00
func get_base_reward_per_increment*(
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
total_active_balance: Gwei): Gwei =
get_base_reward_per_increment_sqrt(integer_squareroot(total_active_balance))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#get_base_reward
cache base_reward_per_increment for 5x+ Altair block_sim perf increase (#2681)
2021-06-28 06:54:06 +00:00
func get_base_reward(
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState,
index: ValidatorIndex, base_reward_per_increment: Gwei): Gwei =
cache base_reward_per_increment for 5x+ Altair block_sim perf increase (#2681)
2021-06-28 06:54:06 +00:00
## Return the base reward for the validator defined by ``index`` with respect
## to the current ``state``.
let increments =
state.validators[index].effective_balance div EFFECTIVE_BALANCE_INCREMENT
increments * base_reward_per_increment
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/phase0/beacon-chain.md#attestations
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
proc check_attestation*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: ForkyBeaconState, attestation: SomeAttestation, flags: UpdateFlags,
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
cache: var StateCache): Result[void, cstring] =
spec updates (#34) * spec updates * make several constants uint64 to help minimize casting * document data type woes - will have to revisit these * change comment style on fields and constants to make room for better comments * add BLSVerify and BLSAddPbkeys facades to insulate spec code from milagro * fix proof of possession type * drop explicitly ordered container fields from ssz - there's an issue open to sort this out before committing it to the spec
2018-12-11 17:55:45 +00:00
## Check that an attestation follows the rules of being included in the state
spec updates (#48) * spec updates * balances move out to separate seq * bunch of placeholders for proof-of-custody / phase1 * fix inclusion distance adjustment * modify state in-place in `updateState` (tests spent over 80% time copying state! now it's down to 25-50) * document several conditions and conversations * some renames here and there to follow spec
2018-12-27 20:14:37 +00:00
## at the current slot. When acting as a proposer, the same rules need to
## be followed!
spec updates (#34) * spec updates * make several constants uint64 to help minimize casting * document data type woes - will have to revisit these * change comment style on fields and constants to make room for better comments * add BLSVerify and BLSAddPbkeys facades to insulate spec code from milagro * fix proof of possession type * drop explicitly ordered container fields from ssz - there's an issue open to sort this out before committing it to the spec
2018-12-11 17:55:45 +00:00
clean up block creation * consistently use state at new block slot to produce block * factor out signature funcs * fix missing block application test
2020-03-19 23:48:03 +00:00
let
data = attestation.data
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
epoch = ? check_attestation_target_epoch(data, state.get_current_epoch())
slot = ? check_attestation_slot_target(data)
committee_index = ? check_attestation_index(
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
data,
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
get_committee_count_per_slot(state, epoch, cache))
? check_attestation_inclusion(slot, state.slot)
Justification/finalization amelioration using testnet1 shard count and epoch length (#331) * working justification, 1 node, 64 validators * closer to tn1 params: 256 validators * more debugging output; switch to minimum test case * working justification and finalization in local network simulation * fix currently incorrect state transition/attestation test assumption
2019-08-14 08:56:32 +00:00
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
let committee_len = get_beacon_committee_len(
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
state, slot, committee_index, cache)
v0.12.2 beacon chain protocol update (#1378)
2020-07-27 10:59:57 +00:00
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
if attestation.aggregation_bits.lenu64 != committee_len:
return err("Inconsistent aggregation and committee length")
Remove get_crosslink_committees_at_slot and fix research/state_sim (#291) * migrate attestation pool tests from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * rm obsolete, unused get_crosslink_committees_at_slot_cached(...) and migrate tests/test_state_transition from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * migrate tests/testutil from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * use more pervasive caching infrastructure, initially of compute_committee; remove buggy (and per-index) shuffling to fix research/state_sim, which was noticing validators on multiple shard committees; rm now-unused specific get_attesting_balance_cached * add some get_active_validator_index caching * rm obsolete/unused get_attesting_indices_cached * rm get_crosslink_committees_at_slot(...) * some more 0.7 changes -- some of which can't be completed pending some data structure updates -- and shard handling changes to offset with epoch start shards
2019-06-24 09:21:56 +00:00
Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex (#3259) * Harden CommitteeIndex, SubnetId, SyncSubcommitteeIndex Harden the use of `CommitteeIndex` et al to prevent future issues by using a distinct type, then validating before use in several cases - datatypes in spec are kept simple though so that invalid data still can be read. * fix invalid epoch used in REST `/eth/v1/beacon/states/{state_id}/committees` committee length (could return invalid data) * normalize some variable names * normalize committee index loops * fix `RestAttesterDuty` to use `uint64` for `validator_committee_index` * validate `CommitteeIndex` on ingress in REST API * update rest rules with stricter parsing * better REST serializers * save lots of memory by not using `zip` ...at least a few bytes!
2022-01-08 23:28:49 +00:00
if epoch == get_current_epoch(state):
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
if not (data.source == state.current_justified_checkpoint):
return err("FFG data not matching current justified epoch")
Remove get_crosslink_committees_at_slot and fix research/state_sim (#291) * migrate attestation pool tests from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * rm obsolete, unused get_crosslink_committees_at_slot_cached(...) and migrate tests/test_state_transition from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * migrate tests/testutil from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * use more pervasive caching infrastructure, initially of compute_committee; remove buggy (and per-index) shuffling to fix research/state_sim, which was noticing validators on multiple shard committees; rm now-unused specific get_attesting_balance_cached * add some get_active_validator_index caching * rm obsolete/unused get_attesting_indices_cached * rm get_crosslink_committees_at_slot(...) * some more 0.7 changes -- some of which can't be completed pending some data structure updates -- and shard handling changes to offset with epoch start shards
2019-06-24 09:21:56 +00:00
else:
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
if not (data.source == state.previous_justified_checkpoint):
return err("FFG data not matching previous justified epoch")
Remove get_crosslink_committees_at_slot and fix research/state_sim (#291) * migrate attestation pool tests from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * rm obsolete, unused get_crosslink_committees_at_slot_cached(...) and migrate tests/test_state_transition from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * migrate tests/testutil from get_crosslink_committees_at_slot(...) to get_crosslink_committee(...) * use more pervasive caching infrastructure, initially of compute_committee; remove buggy (and per-index) shuffling to fix research/state_sim, which was noticing validators on multiple shard committees; rm now-unused specific get_attesting_balance_cached * add some get_active_validator_index caching * rm obsolete/unused get_attesting_indices_cached * rm get_crosslink_committees_at_slot(...) * some more 0.7 changes -- some of which can't be completed pending some data structure updates -- and shard handling changes to offset with epoch start shards
2019-06-24 09:21:56 +00:00
attestation processing speedups * avoid creating indexed attestation just to check signatures - above all, don't create it when not checking signatures ;) * avoid pointer op when adding attestation to pool * better iterator for yielding attestations * add metric / log for attestation packing time
2021-04-14 14:43:29 +00:00
? is_valid_indexed_attestation(state, attestation, flags, cache)
0.6.3 updates (#279) * update IndexedAttestation, verify_slashable_attestation/verify_indexed_attestation, and attester slashing processing to 0.6.3 * rm debug scaffolding * rename Attestation.aggregate_signature -> Attestation.signature; convert various references to AttestationData.slot to get_attestation_slot; implement convert_to_indexed; update checkAttestation and processAttestations to 0.6.3; remove spurious assertion in beacon node related to invalid attestations * replace 0.5 get_winning_root_and_participants with 0.6 get_winning_crosslink_and_attesting_indices; update process_crosslinks to 0.6.3 * mark both remaining 0.6.0 spec implementations as 0.6.3 * clear out all remaining spec version 0.6.1 refs * GENESIS_SLOT and GENESIS_EPOCH are 0 * rm 0.5 get_attestation_participants in favor of 0.6 get_attesting_indices * address mratsim's comment * time can be equal to genesis * fix invalid block assertions; those were essentially spurious * allow toBeaconTime to handle time before genesis (in accordance with now(...) which states time can exist before GENESIS)
2019-06-12 07:48:49 +00:00
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
ok()
more v0.2.0 spec updates (#107) * switch to v0.2.0 shuffling scheme * fix some >80 char lines; add get_total_balance helper function (used in epoch processing, etc); note intended removal of repeat_hash and users thereof; add some int_to_bytes and bytes_to_int helpers; more underflow comments; add Gwei type; adjust generate_seed to include additional entropy source; note egregious inefficiencies * minor code cleanup * move some general helper functions to helpers.nim and clarify little-endian comments
2019-02-13 10:26:32 +00:00
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func get_proposer_reward*(state: ForkyBeaconState,
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
attestation: SomeAttestation,
base_reward_per_increment: Gwei,
cache: var StateCache,
epoch_participation: var EpochParticipationFlags): uint64 =
let participation_flag_indices = get_attestation_participation_flag_indices(
state, attestation.data, state.slot - attestation.data.slot)
for index in get_attesting_indices(
state, attestation.data, attestation.aggregation_bits, cache):
for flag_index, weight in PARTICIPATION_FLAG_WEIGHTS:
if flag_index in participation_flag_indices and
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
not has_flag(epoch_participation.item(index), flag_index):
epoch_participation[index] =
add_flag(epoch_participation.item(index), flag_index)
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
# these are all valid; TODO statically verify or do it type-safely
result += get_base_reward(
state, index, base_reward_per_increment) * weight.uint64
Fix #3650 (participation format in BeaconState result is out of spec) (#3776) * Fix #3650 (participation format in BeaconState result is out of spec) * Make EpochParticipationFlags a distinct type
2022-06-20 05:38:56 +00:00
epoch_participation.asHashList.clearCache()
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
let proposer_reward_denominator =
(WEIGHT_DENOMINATOR.uint64 - PROPOSER_WEIGHT.uint64) *
WEIGHT_DENOMINATOR.uint64 div PROPOSER_WEIGHT.uint64
return result div proposer_reward_denominator
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
proc process_attestation*(
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
state: var ForkyBeaconState, attestation: SomeAttestation, flags: UpdateFlags,
avoid recomputing base_reward_per_increment() across attestations (#2704) * avoid recomputing base_reward_per_increment() across attestations * add adapter template to fix nbench process_attestations build * nfuzz also uses process_attestations()
2021-07-06 06:19:06 +00:00
base_reward_per_increment: Gwei, cache: var StateCache):
remove nbench (#3152) Used during initial development of the spec, `nbench` has fallen behind and by and large been superceded by `block_sim`, `state_sim` and `ncli_db bench`.
2021-12-03 14:46:56 +00:00
Result[void, cstring] =
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
# In the spec, attestation validation is mixed with state mutation, so here
# we've split it into two functions so that the validation logic can be
# reused when looking for suitable blocks to include in attestations.
use ForkedHashedBeaconState in StateData (#2634) * use ForkedHashedBeaconState in StateData * fix FAR_FUTURE_EPOCH -> slot overflow; almost always use assign() * avoid stack allocation in maybeUpgradeStateToAltair() * create and use dispatch functions for check_attester_slashing(), check_proposer_slashing(), and check_voluntary_exit() * use getStateRoot() instead of various state.data.hbsPhase0.root * remove withStateVars.hashedState(), which doesn't work as a design anymore * introduce spec/datatypes/altair into beacon_chain_db * fix inefficient codegen for getStateField(largeStateField) * state_transition_slots() doesn't either need/use blocks or runtime presets * combine process_slots(HBS)/state_transition_slots(HBS) which differ only in last-slot htr optimization * getStateField(StateData, ...) was replaced by getStateField(ForkedHashedBeaconState, ...) * fix rollback * switch some state_transition(), process_slots, makeTestBlocks(), etc to use ForkedHashedBeaconState * remove state_transition(phase0.HashedBeaconState) * remove process_slots(phase0.HashedBeaconState) * remove state_transition_block(phase0.HashedBeaconState) * remove unused callWithBS(); separate case expression from if statement * switch back from nested-ref-object construction to (ref Foo)(Bar())
2021-06-11 17:51:46 +00:00
#
# TODO this should be two separate functions, but
# https://github.com/nim-lang/Nim/issues/18202 means that this being called
# by process_operations() in state_transition_block fails that way.
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
let proposer_index = get_beacon_proposer_index(state, cache)
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
if proposer_index.isNone:
adopt Result[void, string] in place of some bool return signatures (#1275) * adopt Result[void, string] in place of some bool return signatures * string -> cstring to reduce memory allocations; ensure all err() strings are constants, with contextual information from higher-level callers * logScope usage fixes * homogenize err() reporting convention * invalid signature in deposit isn't an error
2020-07-03 17:03:14 +00:00
return err("process_attestation: no beacon proposer index and probably no active validators")
render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators (#617) * render get_beacon_proposer_index(...) return value an option and remove overly aggressive assertion formerly causing crashes when missing validators * follow addBlock(...) after refactoring * [skip ci] fix typo
2019-12-04 10:49:59 +00:00
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
? check_attestation(state, attestation, flags, cache)
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# For phase0
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
template addPendingAttestation(attestations: typed) =
# The genericSeqAssign generated by the compiler to copy the attestation
# data sadly is a processing hotspot - the business with the addDefault
# pointer is here simply to work around the poor codegen
var => let in specs/ and tests/ (#3425)
2022-02-20 20:13:06 +00:00
let pa = attestations.addDefault()
Reset cached indices when resetting cache on SSZ read (#2480) * Reset cached indices when resetting cache on SSZ read When deserializing into an existing structure, the cache should be cleared - goes for json also. Also improve error messages.
2021-04-08 10:11:04 +00:00
if pa.isNil:
return err("process_attestation: too many pending attestations")
performance fixes (#2259) * performance fixes * don't mark tree cache as dirty on read-only List accesses * store only blob in memory for keys and signatures, parse blob lazily * compare public keys by blob instead of parsing / converting to raw * compare Eth2Digest using non-constant-time comparison * avoid some unnecessary validator copying This branch will in particular speed up deposit processing which has been slowing down block replay. Pre (mainnet, 1600 blocks): ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3450.269, 0.000, 3450.269, 3450.269, 1, Initialize DB 0.417, 0.822, 0.036, 21.098, 1400, Load block from database 16.521, 0.000, 16.521, 16.521, 1, Load state from database 27.906, 50.846, 8.104, 1507.633, 1350, Apply block 52.617, 37.029, 20.640, 135.938, 50, Apply epoch block ``` Post: ``` 3502.715, 0.000, 3502.715, 3502.715, 1, Initialize DB 0.080, 0.560, 0.035, 21.015, 1400, Load block from database 17.595, 0.000, 17.595, 17.595, 1, Load state from database 15.706, 11.028, 8.300, 107.537, 1350, Apply block 33.217, 12.622, 17.331, 60.580, 50, Apply epoch block ``` * more perf fixes * load EpochRef cache into StateCache more aggressively * point out security concern with public key cache * reuse proposer index from state when processing block * avoid genericAssign in a few more places * don't parse key when signature is unparseable * fix `==` overload for Eth2Digest * preallocate validator list when getting active validators * speed up proposer index calculation a little bit * reuse cache when replaying blocks in ncli_db * avoid a few more copying loops ``` Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3279.158, 0.000, 3279.158, 3279.158, 1, Initialize DB 0.072, 0.357, 0.035, 13.400, 1400, Load block from database 17.295, 0.000, 17.295, 17.295, 1, Load state from database 5.918, 9.896, 0.198, 98.028, 1350, Apply block 15.888, 10.951, 7.902, 39.535, 50, Apply epoch block 0.000, 0.000, 0.000, 0.000, 0, Database block store ``` * clear full balance cache before processing rewards and penalties ``` All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 3947.901, 0.000, 3947.901, 3947.901, 1, Initialize DB 0.124, 0.506, 0.026, 202.370, 363345, Load block from database 97.614, 0.000, 97.614, 97.614, 1, Load state from database 0.186, 0.188, 0.012, 99.561, 357262, Advance slot, non-epoch 14.161, 5.966, 1.099, 395.511, 11524, Advance slot, epoch 1.372, 4.170, 0.017, 276.401, 363345, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database block store ```
2021-01-25 12:04:18 +00:00
assign(pa[].aggregation_bits, attestation.aggregation_bits)
pa[].data = attestation.data
pa[].inclusion_delay = state.slot - attestation.data.slot
pa[].proposer_index = proposer_index.get().uint64
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
# Altair, Bellatrix, and Capella
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
template updateParticipationFlags(epoch_participation: untyped) =
Precise per-component ETH-denominated rewards tracking This is an alternative take on https://github.com/status-im/nimbus-eth2/pull/3107 that aims for more minimal interventions in the spec modules at the expense of duplicating more of the spec logic in ncli_db.
2021-12-29 02:50:49 +00:00
let proposer_reward = get_proposer_reward(
state, attestation, base_reward_per_increment, cache, epoch_participation)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
increase_balance(state, proposer_index.get, proposer_reward)
when state is phase0.BeaconState:
avoid recomputing base_reward_per_increment() across attestations (#2704) * avoid recomputing base_reward_per_increment() across attestations * add adapter template to fix nbench process_attestations build * nfuzz also uses process_attestations()
2021-07-06 06:19:06 +00:00
doAssert base_reward_per_increment == 0.Gwei
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if attestation.data.target.epoch == get_current_epoch(state):
addPendingAttestation(state.current_epoch_attestations)
else:
addPendingAttestation(state.previous_epoch_attestations)
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
elif state is altair.BeaconState or state is bellatrix.BeaconState or
state is capella.BeaconState:
avoid recomputing base_reward_per_increment() across attestations (#2704) * avoid recomputing base_reward_per_increment() across attestations * add adapter template to fix nbench process_attestations build * nfuzz also uses process_attestations()
2021-07-06 06:19:06 +00:00
doAssert base_reward_per_increment > 0.Gwei
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
if attestation.data.target.epoch == get_current_epoch(state):
updateParticipationFlags(state.current_epoch_participation)
else:
updateParticipationFlags(state.previous_epoch_participation)
restore network sim finalization * relax attestation validation when attestation is incoming but make it more strict when adding to block * share attestation validation logic between attestation pool and state transition * remove a bunch of redundant logging * fix potential underflow in attestation delay checking * fix committee used for attestation shard selection when attesting * fix attestation data construction
2019-08-19 16:41:13 +00:00
else:
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
static: doAssert false
spec cleanups (#1379) * add helper for beacon committee length (used for quickly validating attestations) * refactor some attestation checks to do cheap checks early * validate attestation epoch before computing active validator set * clean up documentation / comments * fill state cache on demand
2020-07-27 16:04:44 +00:00
ok()
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#get_next_sync_committee_indices
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
func get_next_sync_committee_keys(
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState):
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
array[SYNC_COMMITTEE_SIZE, ValidatorPubKey] =
add/support TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH (#3303)
2022-01-27 19:52:08 +00:00
## Return the sequence of sync committee indices, with possible duplicates,
## for the next sync committee.
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
# The sync committe depends on seed and effective balance - it can
# thus only be computed for the current epoch of the state, after balance
# updates have been performed
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
let epoch = get_current_epoch(state) + 1
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
const MAX_RANDOM_BYTE = 255
let
active_validator_indices = get_active_validator_indices(state, epoch)
active_validator_count = uint64(len(active_validator_indices))
seed = get_seed(state, epoch, DOMAIN_SYNC_COMMITTEE)
var
i = 0'u64
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
index = 0
res: array[SYNC_COMMITTEE_SIZE, ValidatorPubKey]
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
hash_buffer: array[40, byte]
hash_buffer[0..31] = seed.data
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
while index < SYNC_COMMITTEE_SIZE:
REST cleanups (#3255) * REST cleanups * reject out-of-range committee requests * print all hex values as lower-case * allow requesting state information by head state root * turn `DomainType` into array (follow spec) * `uint_to_bytesXX` -> `uint_to_bytes` (follow spec) * fix wrong dependent root in `/eth/v1/validator/duties/proposer/` * update documentation - `--subscribe-all-subnets` is no longer needed when using the REST interface with validator clients * more fixes * common helpers for dependent block * remove test rules obsoleted by more strict epoch tests * fix trailing commas * Update docs/the_nimbus_book/src/rest-api.md * Update docs/the_nimbus_book/src/rest-api.md Co-authored-by: sacha <sacha@status.im>
2022-01-08 20:06:34 +00:00
hash_buffer[32..39] = uint_to_bytes(uint64(i div 32))
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
let
add/support TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH (#3303)
2022-01-27 19:52:08 +00:00
shuffled_index = compute_shuffled_index(
uint64(i mod active_validator_count), active_validator_count, seed)
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
candidate_index = active_validator_indices[shuffled_index]
random_byte = eth2digest(hash_buffer).data[i mod 32]
effective_balance = state.validators[candidate_index].effective_balance
if effective_balance * MAX_RANDOM_BYTE >= MAX_EFFECTIVE_BALANCE * random_byte:
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
res[index] = state.validators[candidate_index].pubkey
inc index
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
i += 1'u64
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
res
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
update consensus spec ref URLs to v1.2.0 (#4164)
2022-09-23 07:56:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/beacon-chain.md#get_next_sync_committee
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
func get_next_sync_committee*(
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState):
implement forked merge state/block support (#2890) * implement forked state/block support * merge support for containsOrphan; import cleanup; 80-column lines * add merge block header operations and slot sanity fixture * add epoch state transition tests; implement is_valid_gas_limit(), is_merge_block(), is_execution_enabled(), and compute_timestamp_at_slot() * implement process_execution_payload() and add merge deposit operations tests * add merge block sanity tests * add merge case to syncCommitteeParticipants * v1.1.0-beta.5 updates * reduce getTestStates-based memory usage; don't try to REST-serialize ExecutionPayload transactions without underlying support * add execution payload tests; switch var to let in tests/official/
2021-09-27 14:22:58 +00:00
SyncCommittee =
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
## Return the *next* sync committee for a given ``state``.
update to Altair as of v1.1.0-alpha.7 (#2649) * update to Altair as of v1.1.0-alpha.7 * introduce Altair types into attestation pool * avoid allocating/copying pubkeys excessively in get_next_sync_committee()
2021-06-14 17:42:46 +00:00
var res: SyncCommittee
reward accounting for altair+ (#2981) Similar to the existing `RewardInfo`, this PR adds the infrastructure needed to export epoch processing information from altair+. Because accounting is done somewhat differently, the PR uses a fork-specific object to extrct the information in order to make the cost on the spec side low. * RewardInfo -> EpochInfo, ForkedEpochInfo * use array for computing new sync committee * avoid repeated total active balance computations in block processing * simplify proposer index check * simplify epoch transition tests * pre-compute base increment and reuse in epoch processing, and a few other small optimizations This PR introduces the type and does the heavy lifting in terms of refactoring - the tools that use the accounting will need separate PR:s (as well as refinements to the exportred information)
2021-10-13 14:24:36 +00:00
res.pubkeys.data = get_next_sync_committee_keys(state)
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
# see signatures_batch, TODO shouldn't be here
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
# Deposit processing ensures all keys are valid
update to Altair as of v1.1.0-alpha.7 (#2649) * update to Altair as of v1.1.0-alpha.7 * introduce Altair types into attestation pool * avoid allocating/copying pubkeys excessively in get_next_sync_committee()
2021-06-14 17:42:46 +00:00
var attestersAgg: AggregatePublicKey
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
attestersAgg.init(res.pubkeys.data[0].load().get)
update to Altair as of v1.1.0-alpha.7 (#2649) * update to Altair as of v1.1.0-alpha.7 * introduce Altair types into attestation pool * avoid allocating/copying pubkeys excessively in get_next_sync_committee()
2021-06-14 17:42:46 +00:00
for i in 1 ..< res.pubkeys.data.len:
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
attestersAgg.aggregate(res.pubkeys.data[i].load().get)
update to Altair as of v1.1.0-alpha.7 (#2649) * update to Altair as of v1.1.0-alpha.7 * introduce Altair types into attestation pool * avoid allocating/copying pubkeys excessively in get_next_sync_committee()
2021-06-14 17:42:46 +00:00
res.aggregate_pubkey = finish(attestersAgg).toPubKey()
proposed structure for altair (#2323) * proposed structure for hf1 * refactor datatypes.nim into datatypes/{base, phase0, hf1}.nim * hf1 is Altair * some syncing with alpha 2 * adjust epoch processing to disambiguate access to RewardFlags * relocate StateData to stay consistent with meaning phase 0 StateData * passes v1.1.0 alpha 5 SSZ consensus object tests * Altair block header test fixtures work * fix slash_validator() so that Altair attester slashings, proposer slashings, and voluntary exit textures work * deposit operation Altair test fixtures work * slot sanity and all but a couple epoch transition tests switched to Altair * attestation Altair test fixtures work * Altair block sanity test fixtures work * add working altair sync committee tests * improve workarounds for sum-types-across-modules Nim bug; incorporate SignedBeaconBlock root reconstuction to SSZ byte reader
2021-05-28 15:25:58 +00:00
res
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
add beaconcha.in mainnet relay overview to guide (#4217)
2022-10-04 16:45:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/altair/fork.md#upgrading-the-state
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
func translate_participation(
state: var altair.BeaconState,
pending_attestations: openArray[phase0.PendingAttestation]) =
optimize phase 0 -> Altair transition state processing (#2679)
2021-06-25 12:51:25 +00:00
var cache = StateCache()
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
for attestation in pending_attestations:
let
data = attestation.data
inclusion_delay = attestation.inclusion_delay
# Translate attestation inclusion info to flag indices
participation_flag_indices =
get_attestation_participation_flag_indices(state, data, inclusion_delay)
# Apply flags to all attesting validators
for index in get_attesting_indices(
state, data, attestation.aggregation_bits, cache):
for flag_index in participation_flag_indices:
state.previous_epoch_participation[index] =
fix unnecessary HashList/HashArray cache invalidation (#3660) * SSZ `[]` -> `mitem` * `[]` -> `item` immutable access via mutable instance cannot rely on template overloading, and `[]` cannot be a `func` because of special seq handling in compiler.
2022-05-30 13:30:42 +00:00
add_flag(state.previous_epoch_participation.item(index), flag_index)
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
proc -> func, mainly in spec/state transition and adjecent modules (#3405)
2022-02-17 11:53:55 +00:00
func upgrade_to_altair*(cfg: RuntimeConfig, pre: phase0.BeaconState):
ref altair.BeaconState =
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
var
Fix #3650 (participation format in BeaconState result is out of spec) (#3776) * Fix #3650 (participation format in BeaconState result is out of spec) * Make EpochParticipationFlags a distinct type
2022-06-20 05:38:56 +00:00
empty_participation: EpochParticipationFlags
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
inactivity_scores = HashList[uint64, Limit VALIDATOR_REGISTRY_LIMIT]()
optimize phase 0 -> Altair transition state processing (#2679)
2021-06-25 12:51:25 +00:00
doAssert empty_participation.data.setLen(pre.validators.len)
Fix #3650 (participation format in BeaconState result is out of spec) (#3776) * Fix #3650 (participation format in BeaconState result is out of spec) * Make EpochParticipationFlags a distinct type
2022-06-20 05:38:56 +00:00
empty_participation.asHashList.resetCache()
optimize phase 0 -> Altair transition state processing (#2679)
2021-06-25 12:51:25 +00:00
doAssert inactivity_scores.data.setLen(pre.validators.len)
inactivity_scores.resetCache()
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
var => let in specs/ and tests/ (#3425)
2022-02-20 20:13:06 +00:00
let post = (ref altair.BeaconState)(
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
genesis_time: pre.genesis_time,
genesis_validators_root: pre.genesis_validators_root,
slot: pre.slot,
forkedbeaconstate_helpers -> forks (#2772) Simpler module name for stuff that covers forks * check that runtime config matches database state * also include some assorted altair cleanups * use "standard" genesis fork in local testnet to work around missing runtime config support
2021-08-10 20:46:35 +00:00
fork: altairFork(cfg),
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
# History
latest_block_header: pre.latest_block_header,
block_roots: pre.block_roots,
state_roots: pre.state_roots,
historical_roots: pre.historical_roots,
# Eth1
eth1_data: pre.eth1_data,
eth1_data_votes: pre.eth1_data_votes,
eth1_deposit_index: pre.eth1_deposit_index,
# Registry
validators: pre.validators,
balances: pre.balances,
# Randomness
randao_mixes: pre.randao_mixes,
# Slashings
slashings: pre.slashings,
# Attestations
previous_epoch_participation: empty_participation,
current_epoch_participation: empty_participation,
# Finality
justification_bits: pre.justification_bits,
previous_justified_checkpoint: pre.previous_justified_checkpoint,
current_justified_checkpoint: pre.current_justified_checkpoint,
finalized_checkpoint: pre.finalized_checkpoint,
# Inactivity
inactivity_scores: inactivity_scores
)
# Fill in previous epoch participation from the pre state's pending
# attestations
Altair transition tests (#2624) * Working Altair transition tests * with fixed upstream test vectors, remove state root workaround * switch upgrade_to_altair() to returning a reference * remove test_state_transition * fix invalid fork state/block combinations error messages * avoid memory copies by reintroducing state_transition_slots(var SomeHashedBeaconState)
2021-06-04 10:38:00 +00:00
translate_participation(post[], pre.previous_epoch_attestations.asSeq)
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
# Fill in sync committees
# Note: A duplicate committee is assigned for the current and next committee
# at the fork boundary
Altair transition tests (#2624) * Working Altair transition tests * with fixed upstream test vectors, remove state root workaround * switch upgrade_to_altair() to returning a reference * remove test_state_transition * fix invalid fork state/block combinations error messages * avoid memory copies by reintroducing state_transition_slots(var SomeHashedBeaconState)
2021-06-04 10:38:00 +00:00
post[].current_sync_committee = get_next_sync_committee(post[])
post[].next_sync_committee = get_next_sync_committee(post[])
add Altair fork tests for upgrade_to_altair() (#2620) * add Altair fork tests for upgrade_to_altair() * Deposit processing ensures all keys are valid
2021-05-31 12:54:53 +00:00
post
add merge SSZ consensus object tests (#2858) * add merge SSZ consensus object tests * add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures * throw catchable exception rather than assert on invalid SingleMemberUnion selector * hash_tree_root can assert, because it's trusted data by that point * re-assert on writing, since also trusted data * beta.4 update * implement upgrade_to_merge()
2021-09-11 08:01:05 +00:00
add beaconcha.in mainnet relay overview to guide (#4217)
2022-10-04 16:45:06 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.2.0/specs/bellatrix/fork.md#upgrading-the-state
rename upgrade_to_{merge,bellatrix}; detect unchanging spec YAMLs (#3265)
2022-01-10 09:39:43 +00:00
func upgrade_to_bellatrix*(cfg: RuntimeConfig, pre: altair.BeaconState):
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
ref bellatrix.BeaconState =
add merge SSZ consensus object tests (#2858) * add merge SSZ consensus object tests * add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures * throw catchable exception rather than assert on invalid SingleMemberUnion selector * hash_tree_root can assert, because it's trusted data by that point * re-assert on writing, since also trusted data * beta.4 update * implement upgrade_to_merge()
2021-09-11 08:01:05 +00:00
let epoch = get_current_epoch(pre)
initial migration from spec/datatypes/{merge => bellatrix} (#3249)
2022-01-06 11:25:35 +00:00
(ref bellatrix.BeaconState)(
add merge SSZ consensus object tests (#2858) * add merge SSZ consensus object tests * add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures * throw catchable exception rather than assert on invalid SingleMemberUnion selector * hash_tree_root can assert, because it's trusted data by that point * re-assert on writing, since also trusted data * beta.4 update * implement upgrade_to_merge()
2021-09-11 08:01:05 +00:00
# Versioning
genesis_time: pre.genesis_time,
genesis_validators_root: pre.genesis_validators_root,
slot: pre.slot,
fork: Fork(
previous_version: pre.fork.current_version,
rename MERGE_FORK_VERSION to BELLATRIX_FORK_VERSION
2022-01-20 08:30:33 +00:00
current_version: cfg.BELLATRIX_FORK_VERSION,
add merge SSZ consensus object tests (#2858) * add merge SSZ consensus object tests * add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures * throw catchable exception rather than assert on invalid SingleMemberUnion selector * hash_tree_root can assert, because it's trusted data by that point * re-assert on writing, since also trusted data * beta.4 update * implement upgrade_to_merge()
2021-09-11 08:01:05 +00:00
epoch: epoch,
),
# History
latest_block_header: pre.latest_block_header,
block_roots: pre.block_roots,
state_roots: pre.state_roots,
historical_roots: pre.historical_roots,
# Eth1
eth1_data: pre.eth1_data,
eth1_data_votes: pre.eth1_data_votes,
eth1_deposit_index: pre.eth1_deposit_index,
# Registry
validators: pre.validators,
balances: pre.balances,
# Randomness
randao_mixes: pre.randao_mixes,
# Slashings
slashings: pre.slashings,
# Participation
previous_epoch_participation: pre.previous_epoch_participation,
current_epoch_participation: pre.current_epoch_participation,
# Finality
justification_bits: pre.justification_bits,
previous_justified_checkpoint: pre.previous_justified_checkpoint,
current_justified_checkpoint: pre.current_justified_checkpoint,
finalized_checkpoint: pre.finalized_checkpoint,
# Inactivity
inactivity_scores: pre.inactivity_scores,
# Sync
current_sync_committee: pre.current_sync_committee,
next_sync_committee: pre.next_sync_committee,
# Execution-layer
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
latest_execution_payload_header: default(bellatrix.ExecutionPayloadHeader)
)
add all missing epoch transition tests (#4269)
2022-10-28 08:02:33 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.3.0-alpha.0/specs/capella/fork.md#upgrading-the-state
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
func upgrade_to_capella*(cfg: RuntimeConfig, pre: bellatrix.BeaconState):
ref capella.BeaconState =
let
epoch = get_current_epoch(pre)
latest_execution_payload_header = capella.ExecutionPayloadHeader(
parent_hash: pre.latest_execution_payload_header.parent_hash,
fee_recipient: pre.latest_execution_payload_header.fee_recipient,
state_root: pre.latest_execution_payload_header.state_root,
receipts_root: pre.latest_execution_payload_header.receipts_root,
logs_bloom: pre.latest_execution_payload_header.logs_bloom,
prev_randao: pre.latest_execution_payload_header.prev_randao,
block_number: pre.latest_execution_payload_header.block_number,
gas_limit: pre.latest_execution_payload_header.gas_limit,
gas_used: pre.latest_execution_payload_header.gas_used,
timestamp: pre.latest_execution_payload_header.timestamp,
extra_data: pre.latest_execution_payload_header.extra_data,
base_fee_per_gas: pre.latest_execution_payload_header.base_fee_per_gas,
block_hash: pre.latest_execution_payload_header.block_hash,
transactions_root: pre.latest_execution_payload_header.transactions_root,
withdrawals_root: Eth2Digest() # [New in Capella]
)
(ref capella.BeaconState)(
# Versioning
genesis_time: pre.genesis_time,
genesis_validators_root: pre.genesis_validators_root,
slot: pre.slot,
fork: Fork(
previous_version: pre.fork.current_version,
current_version: cfg.CAPELLA_FORK_VERSION,
epoch: epoch,
),
# History
latest_block_header: pre.latest_block_header,
block_roots: pre.block_roots,
state_roots: pre.state_roots,
historical_roots: pre.historical_roots,
# Eth1
eth1_data: pre.eth1_data,
eth1_data_votes: pre.eth1_data_votes,
eth1_deposit_index: pre.eth1_deposit_index,
# Registry
validators: pre.validators,
balances: pre.balances,
# Randomness
randao_mixes: pre.randao_mixes,
# Slashings
slashings: pre.slashings,
# Participation
previous_epoch_participation: pre.previous_epoch_participation,
current_epoch_participation: pre.current_epoch_participation,
# Finality
justification_bits: pre.justification_bits,
previous_justified_checkpoint: pre.previous_justified_checkpoint,
current_justified_checkpoint: pre.current_justified_checkpoint,
finalized_checkpoint: pre.finalized_checkpoint,
# Inactivity
inactivity_scores: pre.inactivity_scores,
# Sync
current_sync_committee: pre.current_sync_committee,
next_sync_committee: pre.next_sync_committee,
# Execution-layer
latest_execution_payload_header: latest_execution_payload_header,
# Withdrawals
withdrawal_queue: HashList[Withdrawal, Limit WITHDRAWAL_QUEUE_LIMIT](),
next_withdrawal_index: 0,
next_partial_withdrawal_validator_index: 0
add merge SSZ consensus object tests (#2858) * add merge SSZ consensus object tests * add merge attestations, attester slashing, proposer slashing, and voluntary exit test fixtures * throw catchable exception rather than assert on invalid SingleMemberUnion selector * hash_tree_root can assert, because it's trusted data by that point * re-assert on writing, since also trusted data * beta.4 update * implement upgrade_to_merge()
2021-09-11 08:01:05 +00:00
)
Correct implementation of the /validator/duties/sync/{epoch} API According to the spec, this call should return the positions of the specified validators within the sync committee. The existing code was instead returning the indices of the sync sub-committees where the validator is a member.
2021-11-07 21:03:23 +00:00
Address review comments in #3057 (#3069) * Address review comments in #3057 * reorder imports in rest_utils maybe this will help with the mysterious serialization issues Co-authored-by: Jacek Sieka <jacek@status.im>
2021-11-09 19:21:36 +00:00
template isValidInState*(idx: ValidatorIndex, state: ForkyBeaconState): bool =
idx.uint64 < state.validators.lenu64
Correct implementation of the /validator/duties/sync/{epoch} API According to the spec, this call should return the positions of the specified validators within the sync committee. The existing code was instead returning the indices of the sync sub-committees where the validator is a member.
2021-11-07 21:03:23 +00:00
ncli_db: add putState, putBlock (#3096) * ncli_db: add putState, putBlock These tools allow modifying an existing nimbus database for the purpose of recovery or reorg, moving the head, tail and genesis to arbitrary points. * remove potentially expensive `putState` in `BeaconStateDB` * introduce `latest_block_root` which computes the root of the latest applied block from the `latest_block_header` field (instead of passing it in separately) * avoid some unnecessary BeaconState copies during init * discover https://github.com/nim-lang/Nim/issues/19094 * prefer `HashedBeaconState` in a few places to avoid recomputing state root * fetch latest block root from state when creating blocks * harden `get_beacon_proposer_index` against invalid slots and document * move random spec function tests to `test_spec.nim` * avoid unnecessary state root computation before block proposal
2021-11-18 12:02:43 +00:00
func latest_block_root*(state: ForkyBeaconState, state_root: Eth2Digest): Eth2Digest =
# The root of the last block that was successfully applied to this state -
# normally, when a block is applied, the data from the header is stored in
# the state without the state root - on the next process_slot, the state root
# is added to the header and the block root can now be computed and added to
# the block roots table. If process_slot has not yet run on top of the new
# block, we must fill in the state root ourselves.
if state.slot == state.latest_block_header.slot:
# process_slot will not yet have updated the header of the "current" block -
# similar to block creation, we fill it in with the state root
var tmp = state.latest_block_header
tmp.state_root = state_root
hash_tree_root(tmp)
elif state.slot <=
(state.latest_block_header.slot + SLOTS_PER_HISTORICAL_ROOT):
# block_roots is limited to about a day - see assert in
# `get_block_root_at_slot`
state.get_block_root_at_slot(state.latest_block_header.slot)
else:
# Reallly long periods of empty slots - unlikely but possible
hash_tree_root(state.latest_block_header)
func latest_block_root*(state: ForkyHashedBeaconState): Eth2Digest =
latest_block_root(state.data, state.root)
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
func latest_block_root*(state: ForkedHashedBeaconState): Eth2Digest =
more `withState` `state` -> `forkyState` (#4112)
2022-09-13 11:53:12 +00:00
withState(state): latest_block_root(forkyState)
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
func get_sync_committee_cache*(
capella test vector support (#4261)
2022-10-27 06:29:24 +00:00
state: altair.BeaconState | bellatrix.BeaconState | capella.BeaconState,
cache: var StateCache): SyncCommitteeCache =
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
let period = state.slot.sync_committee_period()
cache.sync_committees.withValue(period, v) do:
return v[]
var
s = toHashSet(state.current_sync_committee.pubkeys.data)
for pk in state.next_sync_committee.pubkeys.data:
s.incl(pk)
var pubkeyIndices: Table[ValidatorPubKey, ValidatorIndex]
document and clean up `ValidatorIndex` usage (#3651) * document static vs dynamic range checking requirements * add `vindices` iterator to iterate over valid validator indices in a state * clean up spec comments in general * fixup Co-authored-by: tersec <tersec@users.noreply.github.com>
2022-05-23 23:39:08 +00:00
for vidx in state.validators.vindices:
let pubkey = state.validators[vidx].pubkey
if pubkey in s:
pubkeyIndices[pubkey] = vidx
Speed up altair block processing 2x (#3115) * Speed up altair block processing >2x Like #3089, this PR drastially speeds up historical REST queries and other long state replays. * cache sync committee validator indices * use ~80mb less memory for validator pubkey mappings * batch-verify sync aggregate signature (fixes #2985) * document sync committee hack with head block vs sync message block * add batch signature verification failure tests Before: ``` ../env.sh nim c -d:release -r ncli_db --db:mainnet_0/db bench --start-slot:-1000 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 5830.675, 0.000, 5830.675, 5830.675, 1, Initialize DB 0.481, 1.878, 0.215, 59.167, 981, Load block from database 8422.566, 0.000, 8422.566, 8422.566, 1, Load state from database 6.996, 1.678, 0.042, 14.385, 969, Advance slot, non-epoch 93.217, 8.318, 84.192, 122.209, 32, Advance slot, epoch 20.513, 23.665, 11.510, 201.561, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` After: ``` 7081.422, 0.000, 7081.422, 7081.422, 1, Initialize DB 0.553, 2.122, 0.175, 66.692, 981, Load block from database 5439.446, 0.000, 5439.446, 5439.446, 1, Load state from database 6.829, 1.575, 0.043, 12.156, 969, Advance slot, non-epoch 94.716, 2.749, 88.395, 100.026, 32, Advance slot, epoch 11.636, 23.766, 4.889, 205.250, 981, Apply block, no slot processing 0.000, 0.000, 0.000, 0.000, 0, Database load 0.000, 0.000, 0.000, 0.000, 0, Database store ``` * add comment
2021-11-24 12:43:50 +00:00
var res: SyncCommitteeCache
try:
for i in 0..<res.current_sync_committee.len():
res.current_sync_committee[i] =
pubkeyIndices[state.current_sync_committee.pubkeys[i]]
res.next_sync_committee[i] =
pubkeyIndices[state.next_sync_committee.pubkeys[i]]
except KeyError:
raiseAssert "table constructed just above"
cache.sync_committees[period] = res
res
move dependent root computations to `BeaconState` / `EpochRef` (#3478) * fewer deps on `BlockRef` traversal in anticipation of pruning * allows identifying EpochRef:s by their shuffling as a first step of * tighten error handling around missing blocks using the zero hash for signalling "missing block" is fragile and easy to miss - with checkpoint sync now, and pruning in the future, missing blocks become "normal".
2022-03-15 08:24:55 +00:00
func dependent_root*(state: ForkyHashedBeaconState, epoch: Epoch): Eth2Digest =
## Return the root of the last block that contributed to the shuffling in the
## given epoch
if epoch > state.data.slot.epoch:
state.latest_block_root
elif epoch == Epoch(0):
if state.data.slot == Slot(0):
state.latest_block_root
else:
state.data.get_block_root_at_slot(Slot(0))
else:
let dependent_slot = epoch.start_slot - 1
if state.data.slot <= dependent_slot + SLOTS_PER_HISTORICAL_ROOT:
state.data.get_block_root_at_slot(epoch.start_slot - 1)
else:
Eth2Digest() # "don't know"
func proposer_dependent_root*(state: ForkyHashedBeaconState): Eth2Digest =
state.dependent_root(state.data.slot.epoch)
func attester_dependent_root*(state: ForkyHashedBeaconState): Eth2Digest =
fix order of current/previous dependent root in REST SSE (#4141)
2022-09-19 10:28:52 +00:00
state.dependent_root(state.data.slot.epoch.get_previous_epoch)
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
move `BlockId` to `spec` (#3511) The spec implicitly talks about the slot of a block in several places, and keeping it readily available is useful in a number of context - might as well put this implicitly refereneced helper in the spec code directly
2022-03-16 15:00:18 +00:00
func latest_block_id*(state: ForkyHashedBeaconState): BlockId =
## Block id of the latest block applied to this state
BlockId(
root: state.latest_block_root,
slot: state.data.latest_block_header.slot)
func latest_block_id*(state: ForkedHashedBeaconState): BlockId =
## Block id of the latest block applied to this state
more `withState` `state` -> `forkyState` (#4112)
2022-09-13 11:53:12 +00:00
withState(state): forkyState.latest_block_id()
move `BlockId` to `spec` (#3511) The spec implicitly talks about the slot of a block in several places, and keeping it readily available is useful in a number of context - might as well put this implicitly refereneced helper in the spec code directly
2022-03-16 15:00:18 +00:00
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
func matches_block*(
state: ForkyHashedBeaconState, block_root: Eth2Digest): bool =
## Return true iff the latest block applied to this state matches the given
## `block_root`
block_root == state.latest_block_root
move `BlockId` to `spec` (#3511) The spec implicitly talks about the slot of a block in several places, and keeping it readily available is useful in a number of context - might as well put this implicitly refereneced helper in the spec code directly
2022-03-16 15:00:18 +00:00
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
func matches_block*(
state: ForkedHashedBeaconState, block_root: Eth2Digest): bool =
more `withState` `state` -> `forkyState` (#4112)
2022-09-13 11:53:12 +00:00
withState(state): forkyState.matches_block(block_root)
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
func matches_block_slot*(
state: ForkyHashedBeaconState, block_root: Eth2Digest, slot: Slot): bool =
## Return true iff the latest block applied to this state matches the given
## `block_root` and the state slot has been advanced to the given slot
slot == state.data.slot and block_root == state.latest_block_root
func matches_block_slot*(
state: ForkedHashedBeaconState, block_root: Eth2Digest, slot: Slot): bool =
more `withState` `state` -> `forkyState` (#4112)
2022-09-13 11:53:12 +00:00
withState(state): forkyState.matches_block_slot(block_root, slot)
remove StateData (#3507) One more step on the journey to reduce `BlockRef` usage across the codebase - this one gets rid of `StateData` whose job was to keep track of which block was last assigned to a state - these duties have now been taken over by `latest_block_root`, a fairly recent addition that computes this block root from state data (at a small cost that should be insignificant) 99% mechanical change.
2022-03-16 07:20:40 +00:00
func can_advance_slots*(
state: ForkyHashedBeaconState, block_root: Eth2Digest, target_slot: Slot): bool =
## Return true iff we can reach the given block/slot combination simply by
## advancing slots
target_slot >= state.data.slot and block_root == state.latest_block_root
func can_advance_slots*(
state: ForkedHashedBeaconState, block_root: Eth2Digest, target_slot: Slot): bool =
more `withState` `state` -> `forkyState` (#4112)
2022-09-13 11:53:12 +00:00
withState(state): forkyState.can_advance_slots(block_root, target_slot)