Syncing ======= Syncing blocks is performed in two partially overlapping phases * loading the header chains into separate database tables * removing headers from the headers chain, fetching the rest of the block the header belongs to and executing it Header chains ------------- The header chains are the triple of * a consecutively linked chain of headers starting starting at Genesis * followed by a sequence of missing headers * followed by a consecutively linked chain of headers ending up at a finalised block header received from the consensus layer A sequence *@[h(1),h(2),..]* of block headers is called a consecutively linked chain if * block numbers join without gaps, i.e. *h(n).number+1 == h(n+1).number* * parent hashes match, i.e. *h(n).hash == h(n+1).parentHash* General header chains layout diagram G B L F (1) o----------------o---------------------o----------------o---> | <-- linked --> | <-- unprocessed --> | <-- linked --> | Here, the single upper letter symbols *G*, *B*, *L*, *F* denote block numbers. For convenience, these letters are also identified with its associated block header or the full block. Saying *"the header G"* is short for *"the header with block number G"*. Meaning of *G*, *B*, *L*, *F*: * *G* -- Genesis block number *#0* * *B* -- base, maximal block number of linked chain starting at *G* * *L* -- least, minimal block number of linked chain ending at *F* with *B <= L* * *F* -- final, some finalised block This definition implies *G <= B <= L <= F* and the header chains can uniquely be described by the triple of block numbers *(B,L,F)*. ### Storage of header chains: Some block numbers from the set *{w|G<=w<=B}* may correspond to finalised blocks which may be stored anywhere. If some block numbers do not correspond to finalised blocks, then the headers must reside in the *flareHeader* database table. Of course, due to being finalised such block numbers constitute a sub-chain starting at *G*. The block numbers from the set *{w|L<=w<=F}* must reside in the *flareHeader* database table. They do not correspond to finalised blocks. ### Header chains initialisation: Minimal layout on a pristine system G (2) B L F o---> When first initialised, the header chains are set to *(G,G,G)*. ### Updating header chains: A header chain with an non empty open interval *(B,L)* can be updated only by increasing *B* or decreasing *L* by adding headers so that the linked chain condition is not violated. Only when the open interval *(B,L)* vanishes the right end *F* can be increased by *Z* say. Then * *B==L* beacuse interval *(B,L)* is empty * *B==F* because *B* is maximal and the header chains *(F,F,F)* (depicted in *(3)*) can be set to *(B,Z,Z)* (as depicted in *(4)*.) Layout before updating of *F* B (3) L G F Z o----------------o---------------------o----> | <-- linked --> | New layout with *Z* L' (4) G B F' o----------------o---------------------o----> | <-- linked --> | <-- unprocessed --> | with *L'=Z* and *F'=Z*. Note that diagram *(3)* is a generalisation of *(2)*. ### Complete header chain: The header chain is *relatively complete* if it satisfies clause *(3)* above for *G < B*. It is *fully complete* if *F==Z*. It should be obvious that the latter condition is temporary only on a live system (as *Z* is permanently updated.) If a *relatively complete* header chain is reached for the first time, the execution layer can start running an importer in the background compiling or executing blocks (starting from block number *#1*.) So the ledger database state will be updated incrementally. Imported block chain -------------------- The following imported block chain diagram amends the layout *(1)*: G T B L F (5) o------------------o-------o---------------------o----------------o--> | <-- imported --> | | | | | <------- linked ------> | <-- unprocessed --> | <-- linked --> | where *T* is the number of the last imported and executed block. Coincidentally, *T* also refers to the global state of the ledger database. The headers corresponding to the half open interval `(T,B]` can be completed by fetching block bodies and then imported/executed. Running the sync process for *MainNet* -------------------------------------- For syncing, a beacon node is needed that regularly informs via *RPC* of a recently finalised block header. The beacon node program used here is the *nimbus_beacon_node* binary from the *nimbus-eth2* project (any other will do.) *Nimbus_beacon_node* is started as ./run-mainnet-beacon-node.sh \ --web3-url=http://127.0.0.1:8551 \ --jwt-secret=/tmp/jwtsecret where *http://127.0.0.1:8551* is the URL of the sync process that receives the finalised block header (here on the same physical machine) and `/tmp/jwtsecret` is the shared secret file needed for mutual communication authentication. It will take a while for *nimbus_beacon_node* to catch up (see the [Nimbus Guide](https://nimbus.guide/quick-start.html) for details.) ### Starting `nimbus` for syncing As the sync process is quite slow, it makes sense to pre-load the database with data from an `Era1` archive (if available) before starting the real sync process. The command would be something like ./build/nimbus import \ --era1-dir:/path/to/main-era1/repo \ ... which will take a while for the full *MainNet* era1 repository (but way faster than the sync.) On a system with memory considerably larger than *8GiB* the *nimbus* binary is started on the same machine where the beacon node runs as ./build/nimbus \ --network=mainnet \ --sync-mode=flare \ --engine-api=true \ --engine-api-port=8551 \ --engine-api-ws=true \ --jwt-secret=/tmp/jwtsecret \ ... Note that *--engine-api-port=8551* and *--jwt-secret=/tmp/jwtsecret* match the corresponding options from the *nimbus-eth2* beacon source example. ### Syncing on a low memory machine On a system with memory with *8GiB* the following additional options proved useful for *nimbus* to reduce the memory footprint. For the *Era1* pre-load (if any) the following extra options apply to "*nimbus import*": --chunk-size=1024 --debug-rocksdb-row-cache-size=512000 --debug-rocksdb-block-cache-size=1500000 To start syncing, the following additional options apply to *nimbus*: --debug-flare-chunk-size=384 --debug-rocksdb-max-open-files=384 --debug-rocksdb-write-buffer-size=50331648 --debug-rocksdb-block-cache-size=1073741824 --debug-rdb-key-cache-size=67108864 --debug-rdb-vtx-cache-size=268435456 Also, to reduce the backlog for *nimbus-eth2* stored on disk, the following changes might be considered. For file *nimbus-eth2/vendor/mainnet/metadata/config.yaml* change setting constants: MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024 MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096 to MIN_EPOCHS_FOR_BLOCK_REQUESTS: 8 MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 8 Caveat: These changes are not useful when running *nimbus_beacon_node* as a production system. Metrics ------- The following metrics are defined in *worker/update/metrics.nim* which will be available if *nimbus* is compiled with the additional make flags *NIMFLAGS="-d:metrics \-\-threads:on"*: | *Variable* | *Logic type* | *Short description* | |:------------------------------|:------------:|:--------------------| | | | | | flare_state_block_number | block height | **T**, *increasing* | | flare_base_block_number | block height | **B**, *increasing* | | flare_least_block_number | block height | **L** | | flare_final_block_number | block height | **F**, *increasing* | | flare_beacon_block_number | block height | **Z**, *increasing* | | | | | | flare_headers_staged_queue_len| size | # of staged header list records | | flare_headers_unprocessed | size | # of accumulated header block numbers| | flare_blocks_staged_queue_len | size | # of staged block list records | | flare_blocks_unprocessed | size | # of accumulated body block numbers | | | | | | flare_number_of_buddies | size | # of working peers |