4207b127f9
* era: load blocks and states Era files contain finalized history and can be thought of as an alternative source for block and state data that allows clients to avoid syncing this information from the P2P network - the P2P network is then used to "top up" the client with the most recent data. They can be freely shared in the community via whatever means (http, torrent, etc) and serve as a permanent cold store of consensus data (and, after the merge, execution data) for history buffs and bean counters alike. This PR gently introduces support for loading blocks and states in two cases: block requests from rest/p2p and frontfilling when doing checkpoint sync. The era files are used as a secondary source if the information is not found in the database - compared to the database, there are a few key differences: * the database stores the block indexed by block root while the era file indexes by slot - the former is used only in rest, while the latter is used both by p2p and rest. * when loading blocks from era files, the root is no longer trivially available - if it is needed, it must either be computed (slow) or cached (messy) - the good news is that for p2p requests, it is not needed * in era files, "framed" snappy encoding is used while in the database we store unframed snappy - for p2p2 requests, the latter requires recompression while the former could avoid it * front-filling is the process of using era files to replace backfilling - in theory this front-filling could happen from any block and front-fills with gaps could also be entertained, but our backfilling algorithm cannot take advantage of this because there's no (simple) way to tell it to "skip" a range. * front-filling, as implemented, is a bit slow (10s to load mainnet): we load the full BeaconState for every era to grab the roots of the blocks - it would be better to partially load the state - as such, it would also be good to be able to partially decompress snappy blobs * lookups from REST via root are served by first looking up a block summary in the database, then using the slot to load the block data from the era file - however, there needs to be an option to create the summary table from era files to fully support historical queries To test this, `ncli_db` has an era file exporter: the files it creates should be placed in an `era` folder next to `db` in the data directory. What's interesting in particular about this setup is that `db` remains as the source of truth for security purposes - it stores the latest synced head root which in turn determines where a node "starts" its consensus participation - the era directory however can be freely shared between nodes / people without any (significant) security implications, assuming the era files are consistent / not broken. There's lots of future improvements to be had: * we can drop the in-memory `BlockRef` index almost entirely - at this point, resident memory usage of Nimbus should drop to a cool 500-600 mb * we could serve era files via REST trivially: this would drop backfill times to whatever time it takes to download the files - unlike the current implementation that downloads block by block, downloading an era at a time almost entirely cuts out request overhead * we can "reasonably" recreate detailed state history from almost any point in time, turning an O(slot) process into O(1) effectively - we'll still need caches and indices to do this with sufficient efficiency for the rest api, but at least it cuts the whole process down to minutes instead of hours, for arbitrary points in time * CI: ignore failures with Nim-1.6 (temporary) * test fixes Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com> |
||
|---|---|---|
| .. | ||
| README.md | ||
| request_manager.nim | ||
| sync_manager.nim | ||
| sync_protocol.nim | ||
| sync_queue.nim | ||
README.md
Block syncing
This folder holds all modules related to block syncing
Block syncing uses ETH2 RPC protocol.
Reference diagram
Eth2 RPC in
Blocks are requested during sync by the SyncManager.
Blocks are received by batch:
syncStep(SyncManager, index, peer)- in case of success:
push(SyncQueue, SyncRequest, seq[SignedBeaconBlock]) is called to handle a successful sync step. It callsvalidate(SyncQueue, SignedBeaconBlock)` on each block retrieved one-by-onevalidateonly enqueues the block in the SharedBlockQueueAsyncQueue[BlockEntry]but does no extra validation only the GossipSub case
- in case of failure:
push(SyncQueue, SyncRequest)is called to reschedule the sync request.
Every second when sync is not in progress, the beacon node will ask the RequestManager to download all missing blocks currently in quarantaine.
- via
handleMissingBlocks - which calls
fetchAncestorBlocks - which asynchronously enqueue the request in the SharedBlockQueue
AsyncQueue[BlockEntry].
The RequestManager runs an event loop:
- that calls
fetchAncestorBlocksFromNetwork - which RPC calls peers with
beaconBlocksByRoot - and calls
validate(RequestManager, SignedBeaconBlock)on each block retrieved one-by-one validateonly enqueues the block in theAsyncQueue[BlockEntry]but does no extra validation only the GossipSub case
Weak subjectivity sync
Not implemented!
Comments
The validate procedure name for SyncManager and RequestManager
as no P2P validation actually occurs.
Sync vs Steady State
During sync:
- The RequestManager is deactivated
- The syncManager is working full speed ahead
- Gossip is deactivated
Bottlenecks during sync
During sync:
- The bottleneck is clearing the SharedBlockQueue
AsyncQueue[BlockEntry]viastoreBlockwhich requires full verification (state transition + cryptography)
Backpressure
The SyncManager handles backpressure by ensuring that
current_queue_slot <= request.slot <= current_queue_slot + sq.queueSize * sq.chunkSize.
- queueSize is -1, unbounded, by default according to comment but all init paths uses 1 (?)
- chunkSize is SLOTS_PER_EPOCH = 32
However the shared AsyncQueue[BlockEntry] itself is unbounded.
Concretely:
- The shared
AsyncQueue[BlockEntry]is bounded for sync - The shared
AsyncQueue[BlockEntry]is unbounded for validated gossip blocks
RequestManager and Gossip are deactivated during sync and so do not contribute to pressure.