322 lines
13 KiB
Markdown
322 lines
13 KiB
Markdown
Beacon Sync
|
|
===========
|
|
|
|
Some definition of terms, and a suggestion of how a beacon sync can be encoded
|
|
providing pseudo code is provided by
|
|
[Beacon Sync](https://notes.status.im/nimbus-merge-first-el?both=#Beacon-sync).
|
|
|
|
In the following, the data domain the Beacon Sync acts upon is explored and
|
|
presented. This leads to an implementation description without the help of
|
|
pseudo code but rather provides a definition of the sync and domain state
|
|
at critical moments.
|
|
|
|
For handling block chain imports and related actions, abstraction methods
|
|
from the `forked_chain` module will be used (abbreviated **FC**.) The **FC**
|
|
entities **base** and **latest** from this module are always printed **bold**.
|
|
|
|
|
|
Sync Logic Outline
|
|
------------------
|
|
|
|
Here is a simplification of the sync process intended to provide a mental
|
|
outline of how it works.
|
|
|
|
In the following block chain layouts, a left position always stands for an
|
|
ancestor of a right one.
|
|
|
|
0------C1 (1)
|
|
|
|
0--------L1 (2)
|
|
\_______H1
|
|
|
|
0------------------C2 (3)
|
|
|
|
0--------------------L2 (4)
|
|
\________H2
|
|
|
|
where
|
|
|
|
* *0* is genesis
|
|
* *C1*, *C2* are the *latest* (aka cursor) entities from the **FC** module
|
|
* *L1*, *L2*, are updated *latest* entities from the **FC** module
|
|
* *H1*, *H2* are block headers (or blocks) that are used as sync targets
|
|
|
|
At stage *(1)*, there is a chain of imported blocks *[0,C1]* (written as
|
|
compact interval of block numbers.)
|
|
|
|
At stage *(2)*, there is a sync request to advance up until block *H1* which
|
|
is then fetched from the network along with its ancestors way back until there
|
|
is an ancestor within the chain of imported blocks *[0,L1]*. The chain *[0,L1]*
|
|
is what the *[0,C1]* has morphed into when the chain of blocks ending at *H1*
|
|
finds its ancestor.
|
|
|
|
At stage *(3)* all blocks recently fetched have now been imported via **FC**.
|
|
In addition to that, there might have been additional imports from other
|
|
entities (e.g. `newPayload`) which has advanced *H1* further to *C2*.
|
|
|
|
Stage *(3)* has become similar to stage *(1)* with *C1* renamed as *C2*, ditto
|
|
for the symbols *L2* and *H2* for stage *(4)*.
|
|
|
|
|
|
Implementation, The Gory Details
|
|
--------------------------------
|
|
|
|
### Description of Sync State
|
|
|
|
The following diagram depicts a most general state view of the sync and the
|
|
*FC* modules and at a given point of time
|
|
|
|
0 L (5)
|
|
o--------------------o
|
|
| <--- imported ---> |
|
|
C D H
|
|
o---------------------o----------------o
|
|
| <-- unprocessed --> | <-- linked --> |
|
|
|
|
where
|
|
|
|
* *C* -- coupler, parent of the left endpoint of the chain of headers or blocks
|
|
to be fetched and imported.
|
|
|
|
* *L* -- **latest**, current value of this entity (with the same name) of the
|
|
**FC** module (i.e. the current value when looked up.) *L* need not
|
|
be a parent of any header of the linked chain `(C,H]` (see below for
|
|
notation). Both *L* and *H* might be heads of different forked chains.
|
|
|
|
* *D* -- dangling, header with the least block number of the linked chain in
|
|
progress ending at *H*. This variable is used to record the download
|
|
state eventually reaching *Y* (for notation *D<<H* see clause *(6)*
|
|
below.)
|
|
|
|
* *H* -- head, sync target header which typically was the value of a *sync to
|
|
new head* request (via RPC)
|
|
|
|
The internal sync state (as opposed to the general state also including the
|
|
state of **FC**) is defined by the triple *(C,D,H)*. Other parameters like *L*
|
|
mentioned in *(5)* are considered ephemeral to the sync state. They are always
|
|
seen by its latest values and not cached by the syncer.
|
|
|
|
There are two order releations and some derivatives used to describe relations
|
|
between headers or blocks.
|
|
|
|
For blocks or headers A and B, A is said less or equal B if the (6)
|
|
block numbers are less or equal. Notation: A <= B.
|
|
|
|
The notation A ~ B stands for A <= B <= A which makes <= an order
|
|
relation (relative to ~ rather than ==). If A ~ B does not hold
|
|
then the notation A !~ B is used.
|
|
|
|
The notation A < B stands for A <= B and A !~ B.
|
|
|
|
The notation B-1 stands for any block or header with block number of
|
|
B less one.
|
|
|
|
|
|
For blocks or headers A and B, writing A <- B stands for the block
|
|
A be parent of B (there can only be one parent of B.)
|
|
|
|
For blocks or headers A and B, A is said ancestor of, or equal to B
|
|
if A == B or there is a non-empty parent lineage A <- X <- Y <-..<- B.
|
|
Notation: A << B (note that << is an equivalence relation.)
|
|
|
|
The compact interval notation [A,B] stands for the set {X|A<<X<<B}
|
|
and the half open interval notation stands for [A,B]-{A} (i.e. the
|
|
interval without the left end point.)
|
|
|
|
Note that *A<<B* implies *A<=B*. Boundary conditions that hold for the
|
|
clause *(5)* diagram are
|
|
|
|
there is a Z in [0,L] with C ~ Z, D is in [C,H] (7)
|
|
|
|
|
|
### Sync Processing
|
|
|
|
Sync starts at an idle state
|
|
|
|
0 H L (8)
|
|
o-----------------o--o
|
|
| <--- imported ---> |
|
|
|
|
where *H<=L* (*H* needs only be known by its block number.) The state
|
|
parameters *C* and *D* are irrelevant here.
|
|
|
|
Following, there will be a request to advance *H* to a new position as
|
|
indicated in the diagram below
|
|
|
|
0 B (9)
|
|
o------------o-------o
|
|
| <--- imported ---> | D
|
|
C H
|
|
o--------------------------------------o
|
|
| <----------- unprocessed ----------> |
|
|
|
|
with a new sync state *(C,H,H)*. The parameter *B* is the **base** entity
|
|
of the **FC** module. The parameter *C* is a placeholder with *C ~ B*. The
|
|
parameter *D* is set to the download start position *H*.
|
|
|
|
The syncer then fetches the header chain *(C,H]* from the network. While
|
|
iteratively fetching headers, the syncer state *(C,D,H)* will only change on
|
|
its second position *D* time after a new header was fetched.
|
|
|
|
Having finished downloading then *C~D-1*. The sync state is *(D-1,D,H)*. One
|
|
will end up with a situation like
|
|
|
|
0 Y L (10)
|
|
o---------------o----o
|
|
| <--- imported ---> |
|
|
C Z H
|
|
o----o---------------------------------o
|
|
| <-------------- linked ------------> |
|
|
|
|
for some *Y* in *[0,L]* and *Z* in *(C,H]* where *Y<<Z* with *L* the **latest**
|
|
entity of the **FC** logic.
|
|
|
|
If there are no such *Y* and *Z*, then *(C,H]* is discarded and sync processing
|
|
restarts at clause *(8)* by resetting the sync state (e.g. to *(0,0,0)*.)
|
|
|
|
Otherwise choose *Y* and *Z* with maximal block number of *Y* so that *Y<-Z*.
|
|
Then complete *(Y,H]==[Z,H]* to a lineage of blocks by downloading missing
|
|
block bodies.
|
|
|
|
Having finished with block bodies, the sync state will be expressed as
|
|
*(Y,Y,H)*. With the choice of the first two entries equal it is indicated that
|
|
the lineage *(Y,H]* is fully populated with blocks.
|
|
|
|
0 Y (11)
|
|
o---------------o----o
|
|
| <--- imported ---> |
|
|
Y H
|
|
o-----------------------------------o
|
|
| <------------ blocks -----------> |
|
|
|
|
The blocks *(Y,H]* will then be imported and executed. While this happens, the
|
|
internal state of the **FC** might change/reset so that further import becomes
|
|
impossible. Even when starting import, the block *Y* might not be in *[0,L]*
|
|
anymore due to some internal reset of the **FC** logic. In any of those
|
|
cases, sync processing restarts at clause *(8)* by resetting the sync state.
|
|
|
|
In case all blocks can be imported, one will will end up at
|
|
|
|
0 Y H L (12)
|
|
o-----------------o---------------------------------o---o
|
|
| <--- imported --------------------------------------> |
|
|
|
|
with *H<<L* for *L* the current value of the **latest** entity of the **FC**
|
|
module.
|
|
|
|
In many cases, *H==L* but there are other actors which also might import blocks
|
|
quickly after finishing import of *H* before formally committing this task. So
|
|
*H* can become ancestor of *L*.
|
|
|
|
Now clause *(12)* is equivalent to clause *(8)*.
|
|
|
|
|
|
Running the sync process for *MainNet*
|
|
--------------------------------------
|
|
|
|
For syncing, a beacon node is needed that regularly informs via *RPC* of a
|
|
recently finalised block header.
|
|
|
|
The beacon node program used here is the *nimbus_beacon_node* binary from the
|
|
*nimbus-eth2* project (any other, e.g.the *light client* will do.)
|
|
*Nimbus_beacon_node* is started as
|
|
|
|
./run-mainnet-beacon-node.sh \
|
|
--web3-url=http://127.0.0.1:8551 \
|
|
--jwt-secret=/tmp/jwtsecret
|
|
|
|
where *http://127.0.0.1:8551* is the URL of the sync process that receives the
|
|
finalised block header (here on the same physical machine) and `/tmp/jwtsecret`
|
|
is the shared secret file needed for mutual communication authentication.
|
|
|
|
It will take a while for *nimbus_beacon_node* to catch up (see the
|
|
[Nimbus Guide](https://nimbus.guide/quick-start.html) for details.)
|
|
|
|
### Starting `nimbus` for syncing
|
|
|
|
As the syncing process is quite slow, it makes sense to pre-load the database
|
|
from an *Era1* archive (if available) before starting the real sync process.
|
|
The command for importing an *Era1* reproitory would be something like
|
|
|
|
./build/nimbus_execution_client import \
|
|
--era1-dir:/path/to/main-era1/repo \
|
|
...
|
|
|
|
which will take its time for the full *MainNet* Era1 repository (but way faster
|
|
than the beacon sync.)
|
|
|
|
On a system with memory considerably larger than *8GiB* the *nimbus* binary is
|
|
started on the same machine where the beacon node runs with the command
|
|
|
|
|
|
./build/nimbus_execution_client \
|
|
--network=mainnet \
|
|
--engine-api=true \
|
|
--engine-api-port=8551 \
|
|
--engine-api-ws=true \
|
|
--jwt-secret=/tmp/jwtsecret \
|
|
...
|
|
|
|
Note that *--engine-api-port=8551* and *--jwt-secret=/tmp/jwtsecret* match
|
|
the corresponding options from the *nimbus-eth2* beacon source example.
|
|
|
|
### Syncing on a low memory machine
|
|
|
|
On a system with memory with *8GiB* the following additional options proved
|
|
useful for *nimbus* to reduce the memory footprint.
|
|
|
|
For the *Era1* pre-load (if any) the following extra options apply to
|
|
"*nimbus import*":
|
|
|
|
--chunk-size=1024
|
|
--debug-rocksdb-row-cache-size=512000
|
|
--debug-rocksdb-block-cache-size=1500000
|
|
|
|
To start syncing, the following additional options apply to *nimbus*:
|
|
|
|
--debug-beacon-chunk-size=384
|
|
--debug-rocksdb-max-open-files=384
|
|
--debug-rocksdb-write-buffer-size=50331648
|
|
--debug-rocksdb-block-cache-size=1073741824
|
|
--debug-rdb-key-cache-size=67108864
|
|
--debug-rdb-vtx-cache-size=268435456
|
|
|
|
Also, to reduce the backlog for *nimbus-eth2* stored on disk, the following
|
|
changes might be considered. In the file
|
|
*nimbus-eth2/vendor/mainnet/metadata/config.yaml* change the folloing
|
|
settings
|
|
|
|
MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024
|
|
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096
|
|
to
|
|
|
|
MIN_EPOCHS_FOR_BLOCK_REQUESTS: 8
|
|
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 8
|
|
|
|
Caveat: These changes are not useful when running *nimbus_beacon_node* as a
|
|
production system.
|
|
|
|
Metrics
|
|
-------
|
|
|
|
The following metrics are defined in *worker/update/metrics.nim* which will
|
|
be available if *nimbus* is compiled with the additional make flags
|
|
*NIMFLAGS="-d:metrics \-\-threads:on"*:
|
|
|
|
| *Variable* | *Logic type* | *Short description* |
|
|
|:-------------------|:------------:|:--------------------|
|
|
| | | |
|
|
| beacon_base | block height | **B**, *increasing* |
|
|
| beacon_latest | block height | **L**, *increasing* |
|
|
| beacon_coupler | block height | **C**, *increasing* |
|
|
| beacon_dangling | block height | **D** |
|
|
| beacon_head | block height | **H**, *increasing* |
|
|
| beacon_target | block height | **T**, *increasing* |
|
|
| | | |
|
|
| beacon_header_lists_staged | size | # of staged header list records |
|
|
| beacon_headers_unprocessed | size | # of accumulated header block numbers|
|
|
| beacon_block_lists_staged | size | # of staged block list records |
|
|
| beacon_blocks_unprocessed | size | # of accumulated body block numbers |
|
|
| | | |
|
|
| beacon_buddies | size | # of peers working concurrently |
|