nimbus-eth2/tests
Jacek Sieka 05ffe7b2bf
Prune `BlockRef` on finalization (#3513)
Up til now, the block dag has been using `BlockRef`, a structure adapted
for a full DAG, to represent all of chain history. This is a correct and
simple design, but does not exploit the linearity of the chain once
parts of it finalize.

By pruning the in-memory `BlockRef` structure at finalization, we save,
at the time of writing, a cool ~250mb (or 25%:ish) chunk of memory
landing us at a steady state of ~750mb normal memory usage for a
validating node.

Above all though, we prevent memory usage from growing proportionally
with the length of the chain, something that would not be sustainable
over time -  instead, the steady state memory usage is roughly
determined by the validator set size which grows much more slowly. With
these changes, the core should remain sustainable memory-wise post-merge
all the way to withdrawals (when the validator set is expected to grow).

In-memory indices are still used for the "hot" unfinalized portion of
the chain - this ensure that consensus performance remains unchanged.

What changes is that for historical access, we use a db-based linear
slot index which is cache-and-disk-friendly, keeping the cost for
accessing historical data at a similar level as before, achieving the
savings at no percievable cost to functionality or performance.

A nice collateral benefit is the almost-instant startup since we no
longer load any large indicies at dag init.

The cost of this functionality instead can be found in the complexity of
having to deal with two ways of traversing the chain - by `BlockRef` and
by slot.

* use `BlockId` instead of `BlockRef` where finalized / historical data
may be required
* simplify clearance pre-advancement
* remove dag.finalizedBlocks (~50:ish mb)
* remove `getBlockAtSlot` - use `getBlockIdAtSlot` instead
* `parent` and `atSlot` for `BlockId` now require a `ChainDAGRef`
instance, unlike `BlockRef` traversal
* prune `BlockRef` parents on finality (~200:ish mb)
* speed up ChainDAG init by not loading finalized history index
* mess up light client server error handling - this need revisiting :)
2022-03-17 17:42:56 +00:00
..
consensus_spec Prune `BlockRef` on finalization (#3513) 2022-03-17 17:42:56 +00:00
fork_choice var => let in specs/ and tests/ (#3425) 2022-02-20 20:13:06 +00:00
fuzzing test fixture cleanups (#2976) 2021-10-12 13:36:52 +02:00
helpers Use nim-ssz-serialization module and rm local ssz code 2021-11-10 13:37:24 +02:00
mocking rename random -> prev_randao in Bellatrix for CL specs v1.1.10 (#3460) 2022-03-03 16:08:14 +00:00
simulation restapi.sh: cleanup on exit (#3328) 2022-01-27 13:03:38 +01:00
slashing_protection Minify slashing protection before SQLite (#3393) 2022-03-04 16:43:34 +02:00
spec_block_processing automated CL spec ref URL updates from v1.1.9 to v1.1.10 (#3455) 2022-03-02 10:00:21 +00:00
spec_epoch_processing var => let in specs/ and tests/ (#3425) 2022-02-20 20:13:06 +00:00
all_tests.nim add option to collect light client data (#3474) 2022-03-11 21:28:10 +01:00
nim.cfg Revert writing backfill root to database (#3215) 2021-12-21 11:40:14 +01:00
test_action_tracker.nim Backfill support for ChainDAG (#3171) 2021-12-13 14:36:06 +01:00
test_attestation_pool.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_beacon_chain_db.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_beacon_time.nim add `start_slot` overload for sync periods (#3469) 2022-03-08 11:38:58 +01:00
test_block_dag.nim move dependent root computations to `BeaconState` / `EpochRef` (#3478) 2022-03-15 09:24:55 +01:00
test_block_processor.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_block_quarantine.nim proc -> func, mainly in spec/state transition and adjecent modules (#3405) 2022-02-17 11:53:55 +00:00
test_blockchain_dag.nim Prune `BlockRef` on finalization (#3513) 2022-03-17 17:42:56 +00:00
test_datatypes.nim remove false OnBlockAdded dependency on phase0 HashedBeaconState (#2661) 2021-06-21 08:35:24 +00:00
test_discovery.nim Peer dialing/kicking system overhaul (#3346) 2022-03-11 10:51:53 +00:00
test_engine_authentication.nim implement --jwt-secret and HS256 JWT/JWS signing for engine API alpha.7 (#3440) 2022-02-27 16:55:02 +00:00
test_eth1_monitor.nim rename random -> prev_randao in Bellatrix for CL specs v1.1.10 (#3460) 2022-03-03 16:08:14 +00:00
test_eth2_ssz_serialization.nim use isZeroMemory for Eth2Digest comparisons (#3386) 2022-02-14 05:26:19 +00:00
test_exit_pool.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_forks.nim rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350) 2022-02-02 14:06:55 +01:00
test_gossip_transition.nim rename Beacon{Block,State}Fork.Merge to Bellatrix; update copyright years (#3240) 2022-01-04 09:45:38 +00:00
test_gossip_validation.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_helpers.nim move `state_transition` to `Result` (#3284) 2022-01-17 12:19:58 +01:00
test_honest_validator.nim dynamic sync committee subscriptions (#3308) 2022-01-24 20:40:59 +00:00
test_interop.nim ncli_db: add putState, putBlock (#3096) 2021-11-18 13:02:43 +01:00
test_keymanager_api.nim Remote KeyManager API and number of fixes/tests for KeyManager API (#3360) 2022-02-07 22:36:09 +02:00
test_keystore.nim var => let in specs/ and tests/ (#3425) 2022-02-20 20:13:06 +00:00
test_keystore_management.nim Remote KeyManager API and number of fixes/tests for KeyManager API (#3360) 2022-02-07 22:36:09 +02:00
test_light_client.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_message_signatures.nim batch-verify sync messages for a small perf boost (#3151) 2021-12-09 14:56:54 +02:00
test_peer_pool.nim proc -> func in ncli/, research/, and test/ (#2818) 2021-08-25 14:51:52 +00:00
test_spec.nim move `BlockId` to `spec` (#3511) 2022-03-16 16:00:18 +01:00
test_statediff.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
test_sync_committee_pool.nim rename no{R=>r}eturn, no{I=>i}init, short{l=>L}og, E{T=>t}h2Node, Beacon{c=>C}hainDB (#3403) 2022-02-16 23:24:44 +01:00
test_sync_manager.nim avoid re-requesting finalized blocks during sync (#3461) 2022-03-15 18:56:56 +01:00
test_zero_signature.nim update spec references from eth2.0-specs to consensus-specs and to v1.1.0-beta.2 (#2822) 2021-08-26 10:21:52 +02:00
testblockutil.nim remove StateData (#3507) 2022-03-16 08:20:40 +01:00
testdbutil.nim Support starting from altair (#3054) 2021-11-10 13:39:08 +02:00
teststateutil.nim rename MERGE_FORK_EPOCH to BELLATRIX_FORK_EPOCH (#3350) 2022-02-02 14:06:55 +01:00
testutil.nim forkedbeaconstate_helpers -> forks (#2772) 2021-08-10 22:46:35 +02:00