status-im/nimbus-eth2
2
0
Fork 0
mirror of https://github.com/status-im/nimbus-eth2.git synced 2025-02-22 19:28:20 +00:00
Code Issues Projects Releases Wiki Activity
nimbus-eth2/beacon_chain/sync
History
Eugene Kabanov ba934af925
Make syncing performance more steady. (#6722) 
* Initial commit.

* Update tests.

* Update AllTests.

* Post-rebase update AllTests.

* Improve log statements which was hidden because of logScope issues.

* Eliminate duplicate `direction` fields.
Debugging blobs map.

* Fix blobs shortmap generation.

* Add fixes to map generation and add test helpers.

* Fix missing dependency.

* Adjust callbacks functionality for untrusted sync.

* Attempt to fix defect state issue with new flag `setLastStateRoot`.

* Make possible fix less intrusive.

* Fix forward declaration missing `=`.

* Revert attempt to improve performance.

* Revert some all the changes.

* Remove spaces.

* Remove LFS fixes from the PR.

* Update AllTests.

* Update copyright year.

* Update AllTests.

* Post-rebase fixes.
2025-02-04 15:36:25 +00:00
..
light_client_manager.nim
automated consensus spec URL updating to v1.5.0-beta.0 (#6833)
2025-01-10 13:10:49 +01:00
light_client_protocol.nim
automated consensus spec URL updating to v1.5.0-beta.0 (#6833)
2025-01-10 13:10:49 +01:00
light_client_sync_helpers.nim
verify genesis_time more strictly (fixes #1667) (#5694)
2024-01-06 15:26:56 +01:00
README.md
docs: fix typos (#5571)
2023-11-06 03:56:07 +00:00
request_manager.nim
update ShufflingRef version of get_attesting_indices to Electra v1.5.0-alpha.10 (#6801)
2024-12-30 09:36:58 +00:00
sync_manager.nim
Make syncing performance more steady. (#6722)
2025-02-04 15:36:25 +00:00
sync_overseer.nim
Make syncing performance more steady. (#6722)
2025-02-04 15:36:25 +00:00
sync_protocol.nim
allow electra blobs per block in req/resp (#6880)
2025-01-28 20:44:41 +00:00
sync_queue.nim
Make syncing performance more steady. (#6722)
2025-02-04 15:36:25 +00:00
sync_types.nim
Light forward sync mechanism (#6515)
2024-10-30 05:38:53 +00:00

README.md

Block syncing

This folder holds all modules related to block syncing

Block syncing uses ETH2 RPC protocol.

Reference diagram

Block flow

Eth2 RPC in

Blocks are requested during sync by the SyncManager.

Blocks are received by batch:

  • syncStep(SyncManager, index, peer)
  • in case of success:
    • push(SyncQueue, SyncRequest, seq[SignedBeaconBlock]) is called to handle a successful sync step. It calls validate(SyncQueue, SignedBeaconBlock)` on each block retrieved one-by-one
    • validate only enqueues the block in the SharedBlockQueue AsyncQueue[BlockEntry] but does no extra validation only the GossipSub case
  • in case of failure:
    • push(SyncQueue, SyncRequest) is called to reschedule the sync request.

Every second when sync is not in progress, the beacon node will ask the RequestManager to download all missing blocks currently in quarantine.

  • via handleMissingBlocks
  • which calls fetchAncestorBlocks
  • which asynchronously enqueue the request in the SharedBlockQueue AsyncQueue[BlockEntry].

The RequestManager runs an event loop:

  • that calls fetchAncestorBlocksFromNetwork
  • which RPC calls peers with beaconBlocksByRoot
  • and calls validate(RequestManager, SignedBeaconBlock) on each block retrieved one-by-one
  • validate only enqueues the block in the AsyncQueue[BlockEntry] but does no extra validation only the GossipSub case

Weak subjectivity sync

Not implemented!

Comments

The validate procedure name for SyncManager and RequestManager as no P2P validation actually occurs.

Sync vs Steady State

During sync:

  • The RequestManager is deactivated
  • The syncManager is working full speed ahead
  • Gossip is deactivated

Bottlenecks during sync

During sync:

  • The bottleneck is clearing the SharedBlockQueue AsyncQueue[BlockEntry] via storeBlock which requires full verification (state transition + cryptography)

Backpressure

The SyncManager handles backpressure by ensuring that current_queue_slot <= request.slot <= current_queue_slot + sq.queueSize * sq.chunkSize.

  • queueSize is -1, unbounded, by default according to comment but all init paths uses 1 (?)
  • chunkSize is SLOTS_PER_EPOCH = 32

However the shared AsyncQueue[BlockEntry] itself is unbounded. Concretely:

  • The shared AsyncQueue[BlockEntry] is bounded for sync
  • The shared AsyncQueue[BlockEntry] is unbounded for validated gossip blocks

RequestManager and Gossip are deactivated during sync and so do not contribute to pressure.