status-im
/
nimbus-eth2
mirror of https://github.com/status-im/nimbus-eth2.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/beacon_chain_db_immutable.nim

198 lines
6.8 KiB
Nim
Raw Normal View History

immutable validator database factoring (#2297) * initial immutable validator database factoring * remove changes from chain_dag: this abstraction properly belongs in beacon_chain_db * add merging mutable/immutable validator portions; individually test database roundtripping of immutable validators and states-sans-immutable-validators * update test summaries * use stew/assign2 instead of Nim assignment * add reading/writing of immutable validators in chaindag * remove unused import * replace chunked k/v store of immutable validators with per-row SQL table storage * use List instead of HashList * un-stub some ncli_db code so that it uses * switch HashArray to array; move BeaconStateNoImmutableValidators from datatypes to beacon_chain_db * begin only-mutable-part state storage * uncomment some assigns * work around https://github.com/nim-lang/Nim/issues/17253 * fix most of the issues/oversights; local sim runs again * fix test suite by adding missing beaconstate field to copy function * have ncli bench also store immutable validators * extract some immutable-validator-specific code from the beacon chain db module * add more rigorous database state roundtripping, with changing validator sets * adjust ncli_db to use new schema * simplify putState/getState by moving all immutable validator accounting into beacon state DB * remove redundant test case and move code to immutable-beacon-chain module * more efficient, but still brute-force, mutable+immutable validator merging * reuse BeaconState in getState * ensure HashList/HashArray caches are cleared when reusing getState buffers; add ncli_db and a unit test to verify this * HashList.clear() -> HashList.clearCache() * only copy incrementally necessary immutable validators * increase strictness of test cases and fix/work around resulting HashList cache invalidation issues * remove explanatory scaffolding * allow for storage of full (with all validators) states for backwards/forwards-compatibility * adjust DbSeq type usage * store full, with-validators, state every 64 epochs to enable reverting versions * reduce memory allocation and intermediate objects in state storage codepath * eliminate allocation/copying through intermediate BeaconStateNoImmutableValidators objects * skip benchmarking initial genesis-validator-heavy state store * always store new-style state and sometimes old-style state * document intent behind BeaconState/Validator type-punnery * more accurate failure message on SQLite in-memory database initialization failure
2021-03-15 14:11:51 +00:00
# beacon_chain
# Copyright (c) 2021 Status Research & Development GmbH
# Licensed and distributed under either of
# * MIT license (license terms in the root directory or at https://opensource.org/licenses/MIT).
# * Apache v2 license (license terms in the root directory or at https://www.apache.org/licenses/LICENSE-2.0).
# at your option. This file may not be copied, modified, or distributed except according to those terms.
{.push raises: [Defect].}
import
Parallel attestation verification (#2718) * Add parallel attestation verification * Update tests, batchVerify doesn't use the threadpool with only single core (nim-blscurve update) * bump nim-blscurve * Debug info for failing eth2 test vectors * remove submodule eth2-testnets * verbose debugging of make failure on Windows (libbacktrace?) * Remove CI debug mode * initialization convention * Fix new altair tests
2021-09-17 00:13:52 +00:00
stew/[assign2, results],
json cleanups (#2456) * move json-rpc specific marshalling to rpc * serialize Epoch/Slot with cast to avoid Defect * avoid a few eth1 deps * simplify imports
2021-03-26 14:11:06 +00:00
serialization,
make isomorphicCast available outside immutable beacon chain database (#2696) * make isomorphicCast available outside immutable beacon chain database
2021-07-02 17:38:14 +00:00
eth/db/kvstore,
merge hardfork database support (#2911) * merge hardfork database support * working block_sim * recreate state transition changes
2021-09-30 01:07:24 +00:00
./spec/datatypes/[base, altair, merge],
disentangle eth2 types from the ssz library (#2785) * reorganize ssz dependencies This PR continues the work in https://github.com/status-im/nimbus-eth2/pull/2646, https://github.com/status-im/nimbus-eth2/pull/2779 as well as past issues with serialization and type, to disentangle SSZ from eth2 and at the same time simplify imports and exports with a structured approach. The principal idea here is that when a library wants to introduce SSZ support, they do so via 3 files: * `ssz_codecs` which imports and reexports `codecs` - this covers the basic byte conversions and ensures no overloads get lost * `xxx_merkleization` imports and exports `merkleization` to specialize and get access to `hash_tree_root` and friends * `xxx_ssz_serialization` imports and exports `ssz_serialization` to specialize ssz for a specific library Those that need to interact with SSZ always import the `xxx_` versions of the modules and never `ssz` itself so as to keep imports simple and safe. This is similar to how the REST / JSON-RPC serializers are structured in that someone wanting to serialize spec types to REST-JSON will import `eth2_rest_serialization` and nothing else. * split up ssz into a core library that is independendent of eth2 types * rename `bytes_reader` to `codec` to highlight that it contains coding and decoding of bytes and native ssz types * remove tricky List init overload that causes compile issues * get rid of top-level ssz import * reenable merkleization tests * move some "standard" json serializers to spec * remove `ValidatorIndex` serialization for now * remove test_ssz_merkleization * add tests for over/underlong byte sequences * fix broken seq[byte] test - seq[byte] is not an SSZ type There are a few things this PR doesn't solve: * like #2646 this PR is weak on how to handle root and other dontSerialize fields that "sometimes" should be computed - the same problem appears in REST / JSON-RPC etc * Fix a build problem on macOS * Another way to fix the macOS builds Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2021-08-18 18:57:58 +00:00
./spec/[eth2_ssz_serialization, eth2_merkleization]
immutable validator database factoring (#2297) * initial immutable validator database factoring * remove changes from chain_dag: this abstraction properly belongs in beacon_chain_db * add merging mutable/immutable validator portions; individually test database roundtripping of immutable validators and states-sans-immutable-validators * update test summaries * use stew/assign2 instead of Nim assignment * add reading/writing of immutable validators in chaindag * remove unused import * replace chunked k/v store of immutable validators with per-row SQL table storage * use List instead of HashList * un-stub some ncli_db code so that it uses * switch HashArray to array; move BeaconStateNoImmutableValidators from datatypes to beacon_chain_db * begin only-mutable-part state storage * uncomment some assigns * work around https://github.com/nim-lang/Nim/issues/17253 * fix most of the issues/oversights; local sim runs again * fix test suite by adding missing beaconstate field to copy function * have ncli bench also store immutable validators * extract some immutable-validator-specific code from the beacon chain db module * add more rigorous database state roundtripping, with changing validator sets * adjust ncli_db to use new schema * simplify putState/getState by moving all immutable validator accounting into beacon state DB * remove redundant test case and move code to immutable-beacon-chain module * more efficient, but still brute-force, mutable+immutable validator merging * reuse BeaconState in getState * ensure HashList/HashArray caches are cleared when reusing getState buffers; add ncli_db and a unit test to verify this * HashList.clear() -> HashList.clearCache() * only copy incrementally necessary immutable validators * increase strictness of test cases and fix/work around resulting HashList cache invalidation issues * remove explanatory scaffolding * allow for storage of full (with all validators) states for backwards/forwards-compatibility * adjust DbSeq type usage * store full, with-validators, state every 64 epochs to enable reverting versions * reduce memory allocation and intermediate objects in state storage codepath * eliminate allocation/copying through intermediate BeaconStateNoImmutableValidators objects * skip benchmarking initial genesis-validator-heavy state store * always store new-style state and sometimes old-style state * document intent behind BeaconState/Validator type-punnery * more accurate failure message on SQLite in-memory database initialization failure
2021-03-15 14:11:51 +00:00
json cleanups (#2456) * move json-rpc specific marshalling to rpc * serialize Epoch/Slot with cast to avoid Defect * avoid a few eth1 deps * simplify imports
2021-03-26 14:11:06 +00:00
type
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157)
2021-12-03 17:40:23 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.6/specs/phase0/beacon-chain.md#beaconstate
introduce database support for Altair (#2667) * introduce immutable Altair BeaconState * add database support for Altair blocks and states * add tests for Altair get/put/contains/delete state * enable blockchain_dag Altair state database storing * properly return error on getting missing altair block
2021-06-24 07:11:47 +00:00
# Memory-representation-equivalent to a phase0 BeaconState for in-place SSZ reading and writing
altair fork handling cleanups (#3050) * fix stack overflow crash in REST/debug/getStateV2 * introduce `ForkyXxx` for generic type matching of `Xxx` across branches (SomeHashedBeaconState -> ForkyHashedBeaconState et al) - `Some` is already used for other types of type classes * consolidate function naming in BeaconChainDB, use some generics * import `forks.nim` from other spec modules and move `Forked*` helpers around to resolve circular imports * remove `ForkedBeaconState`, use `ForkedHashedBeaconState` throughout (less data shuffling between the types) * fix several cases of states being stored on stack in tests, causing random failures on some platforms * remove reading json support from ncli - this should be ported to the rest json reading instead (doesn't currently work because stack sizes)
2021-11-05 07:34:34 +00:00
Phase0BeaconStateNoImmutableValidators* = object
immutable validator database factoring (#2297) * initial immutable validator database factoring * remove changes from chain_dag: this abstraction properly belongs in beacon_chain_db * add merging mutable/immutable validator portions; individually test database roundtripping of immutable validators and states-sans-immutable-validators * update test summaries * use stew/assign2 instead of Nim assignment * add reading/writing of immutable validators in chaindag * remove unused import * replace chunked k/v store of immutable validators with per-row SQL table storage * use List instead of HashList * un-stub some ncli_db code so that it uses * switch HashArray to array; move BeaconStateNoImmutableValidators from datatypes to beacon_chain_db * begin only-mutable-part state storage * uncomment some assigns * work around https://github.com/nim-lang/Nim/issues/17253 * fix most of the issues/oversights; local sim runs again * fix test suite by adding missing beaconstate field to copy function * have ncli bench also store immutable validators * extract some immutable-validator-specific code from the beacon chain db module * add more rigorous database state roundtripping, with changing validator sets * adjust ncli_db to use new schema * simplify putState/getState by moving all immutable validator accounting into beacon state DB * remove redundant test case and move code to immutable-beacon-chain module * more efficient, but still brute-force, mutable+immutable validator merging * reuse BeaconState in getState * ensure HashList/HashArray caches are cleared when reusing getState buffers; add ncli_db and a unit test to verify this * HashList.clear() -> HashList.clearCache() * only copy incrementally necessary immutable validators * increase strictness of test cases and fix/work around resulting HashList cache invalidation issues * remove explanatory scaffolding * allow for storage of full (with all validators) states for backwards/forwards-compatibility * adjust DbSeq type usage * store full, with-validators, state every 64 epochs to enable reverting versions * reduce memory allocation and intermediate objects in state storage codepath * eliminate allocation/copying through intermediate BeaconStateNoImmutableValidators objects * skip benchmarking initial genesis-validator-heavy state store * always store new-style state and sometimes old-style state * document intent behind BeaconState/Validator type-punnery * more accurate failure message on SQLite in-memory database initialization failure
2021-03-15 14:11:51 +00:00
# Versioning
genesis_time*: uint64
genesis_validators_root*: Eth2Digest
slot*: Slot
fork*: Fork
# History
latest_block_header*: BeaconBlockHeader ##\
## `latest_block_header.state_root == ZERO_HASH` temporarily
block_roots*: HashArray[Limit SLOTS_PER_HISTORICAL_ROOT, Eth2Digest] ##\
## Needed to process attestations, older to newer
state_roots*: HashArray[Limit SLOTS_PER_HISTORICAL_ROOT, Eth2Digest]
historical_roots*: HashList[Eth2Digest, Limit HISTORICAL_ROOTS_LIMIT]
# Eth1
eth1_data*: Eth1Data
eth1_data_votes*:
HashList[Eth1Data, Limit(EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH)]
eth1_deposit_index*: uint64
# Registry
validators*: HashList[ValidatorStatus, Limit VALIDATOR_REGISTRY_LIMIT]
balances*: HashList[uint64, Limit VALIDATOR_REGISTRY_LIMIT]
# Randomness
randao_mixes*: HashArray[Limit EPOCHS_PER_HISTORICAL_VECTOR, Eth2Digest]
# Slashings
slashings*: HashArray[Limit EPOCHS_PER_SLASHINGS_VECTOR, uint64] ##\
## Per-epoch sums of slashed effective balances
# Attestations
previous_epoch_attestations*:
HashList[PendingAttestation, Limit(MAX_ATTESTATIONS * SLOTS_PER_EPOCH)]
current_epoch_attestations*:
HashList[PendingAttestation, Limit(MAX_ATTESTATIONS * SLOTS_PER_EPOCH)]
# Finality
justification_bits*: uint8 ##\
## Bit set for every recent justified epoch
## Model a Bitvector[4] as a one-byte uint, which should remain consistent
## with ssz/hashing.
previous_justified_checkpoint*: Checkpoint ##\
## Previous epoch snapshot
current_justified_checkpoint*: Checkpoint
finalized_checkpoint*: Checkpoint
v1.1.{5 -> 6} phase 0 and altair spec URL updates (#3157)
2021-12-03 17:40:23 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.6/specs/altair/beacon-chain.md#beaconstate
introduce database support for Altair (#2667) * introduce immutable Altair BeaconState * add database support for Altair blocks and states * add tests for Altair get/put/contains/delete state * enable blockchain_dag Altair state database storing * properly return error on getting missing altair block
2021-06-24 07:11:47 +00:00
# Memory-representation-equivalent to an Altair BeaconState for in-place SSZ
# reading and writing
AltairBeaconStateNoImmutableValidators* = object
# Versioning
genesis_time*: uint64
genesis_validators_root*: Eth2Digest
slot*: Slot
fork*: Fork
# History
latest_block_header*: BeaconBlockHeader ##\
## `latest_block_header.state_root == ZERO_HASH` temporarily
block_roots*: HashArray[Limit SLOTS_PER_HISTORICAL_ROOT, Eth2Digest] ##\
## Needed to process attestations, older to newer
state_roots*: HashArray[Limit SLOTS_PER_HISTORICAL_ROOT, Eth2Digest]
historical_roots*: HashList[Eth2Digest, Limit HISTORICAL_ROOTS_LIMIT]
# Eth1
eth1_data*: Eth1Data
eth1_data_votes*:
HashList[Eth1Data, Limit(EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH)]
eth1_deposit_index*: uint64
# Registry
validators*: HashList[ValidatorStatus, Limit VALIDATOR_REGISTRY_LIMIT]
balances*: HashList[uint64, Limit VALIDATOR_REGISTRY_LIMIT]
# Randomness
randao_mixes*: HashArray[Limit EPOCHS_PER_HISTORICAL_VECTOR, Eth2Digest]
# Slashings
slashings*: HashArray[Limit EPOCHS_PER_SLASHINGS_VECTOR, uint64] ##\
## Per-epoch sums of slashed effective balances
# Participation
previous_epoch_participation*:
HashList[ParticipationFlags, Limit VALIDATOR_REGISTRY_LIMIT]
current_epoch_participation*:
HashList[ParticipationFlags, Limit VALIDATOR_REGISTRY_LIMIT]
# Finality
justification_bits*: uint8 ##\
## Bit set for every recent justified epoch
## Model a Bitvector[4] as a one-byte uint, which should remain consistent
## with ssz/hashing.
previous_justified_checkpoint*: Checkpoint ##\
## Previous epoch snapshot
current_justified_checkpoint*: Checkpoint
finalized_checkpoint*: Checkpoint
# Inactivity
inactivity_scores*: HashList[uint64, Limit VALIDATOR_REGISTRY_LIMIT] # [New in Altair]
# Light client sync committees
current_sync_committee*: SyncCommittee # [New in Altair]
next_sync_committee*: SyncCommittee # [New in Altair]
merge hardfork database support (#2911) * merge hardfork database support * working block_sim * recreate state transition changes
2021-09-30 01:07:24 +00:00
update 22 spec URLs to v1.1.5 (#3111)
2021-11-18 08:08:00 +00:00
# https://github.com/ethereum/consensus-specs/blob/v1.1.5/specs/merge/beacon-chain.md#beaconstate
merge hardfork database support (#2911) * merge hardfork database support * working block_sim * recreate state transition changes
2021-09-30 01:07:24 +00:00
# Memory-representation-equivalent to a Merge BeaconState for in-place SSZ
# reading and writing
MergeBeaconStateNoImmutableValidators* = object
# Versioning
genesis_time*: uint64
genesis_validators_root*: Eth2Digest
slot*: Slot
fork*: Fork
# History
latest_block_header*: BeaconBlockHeader ##\
## `latest_block_header.state_root == ZERO_HASH` temporarily
block_roots*: HashArray[Limit SLOTS_PER_HISTORICAL_ROOT, Eth2Digest] ##\
## Needed to process attestations, older to newer
state_roots*: HashArray[Limit SLOTS_PER_HISTORICAL_ROOT, Eth2Digest]
historical_roots*: HashList[Eth2Digest, Limit HISTORICAL_ROOTS_LIMIT]
# Eth1
eth1_data*: Eth1Data
eth1_data_votes*:
HashList[Eth1Data, Limit(EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH)]
eth1_deposit_index*: uint64
# Registry
validators*: HashList[ValidatorStatus, Limit VALIDATOR_REGISTRY_LIMIT]
balances*: HashList[uint64, Limit VALIDATOR_REGISTRY_LIMIT]
# Randomness
randao_mixes*: HashArray[Limit EPOCHS_PER_HISTORICAL_VECTOR, Eth2Digest]
# Slashings
slashings*: HashArray[Limit EPOCHS_PER_SLASHINGS_VECTOR, uint64] ##\
## Per-epoch sums of slashed effective balances
# Participation
previous_epoch_participation*:
HashList[ParticipationFlags, Limit VALIDATOR_REGISTRY_LIMIT]
current_epoch_participation*:
HashList[ParticipationFlags, Limit VALIDATOR_REGISTRY_LIMIT]
# Finality
justification_bits*: uint8 ##\
## Bit set for every recent justified epoch
## Model a Bitvector[4] as a one-byte uint, which should remain consistent
## with ssz/hashing.
previous_justified_checkpoint*: Checkpoint ##\
## Previous epoch snapshot
current_justified_checkpoint*: Checkpoint
finalized_checkpoint*: Checkpoint
# Inactivity
inactivity_scores*: HashList[uint64, Limit VALIDATOR_REGISTRY_LIMIT]
# Sync
current_sync_committee*: SyncCommittee
next_sync_committee*: SyncCommittee
# Execution
latest_execution_payload_header*: ExecutionPayloadHeader # [New in Merge]