nimbus-eth1/nimbus/sync/beacon

Beacon Sync

Some definition of terms, and a suggestion of how a beacon sync can be encoded providing pseudo code is provided by Beacon Sync.

In the following, the data domain the Beacon Sync acts upon is explored and presented. This leads to an implementation description without the help of pseudo code but rather provides a definition of the sync and domain state at critical moments.

For handling block chain imports and related actions, abstraction methods from the forked_chain module will be used (abbreviated FC.) The FC entities base and latest from this module are always printed bold.

Sync Logic Outline

Here is a simplification of the sync process intended to provide a mental outline of how it works.

In the following block chain layouts, a left position always stands for an ancestor of a right one.

    0------C1                                                            (1)

    0--------L1                                                          (2)
             \_______H1

    0------------------C2                                                (3)

    0--------------------L2                                              (4)
                        \________H2

where

• 0 is genesis
• C1, C2 are the latest (aka cursor) entities from the FC module
• L1, L2, are updated latest entities from the FC module
• H1, H2 are block headers (or blocks) that are used as sync targets

At stage (1), there is a chain of imported blocks [0,C1] (written as compact interval of block numbers.)

At stage (2), there is a sync request to advance up until block H1 which is then fetched from the network along with its ancestors way back until there is an ancestor within the chain of imported blocks [0,L1]. The chain [0,L1] is what the [0,C1] has morphed into when the chain of blocks ending at H1 finds its ancestor.

At stage (3) all blocks recently fetched have now been imported via FC. In addition to that, there might have been additional imports from other entities (e.g. newPayload) which has advanced H1 further to C2.

Stage (3) has become similar to stage (1) with C1 renamed as C2, ditto for the symbols L2 and H2 for stage (4).

Implementation, The Gory Details

Description of Sync State

The following diagram depicts a most general state view of the sync and the FC modules and at a given point of time

    0            C       L                                               (5)
    o------------o-------o
    | <--- imported ---> |
                 Y                     D                H
                 o---------------------o----------------o
                 | <-- unprocessed --> | <-- linked --> |

where

• C -- coupler, cached base entity of the FC module, reported at the time when H was set. This determines the maximal way back length of the linked ancestor chain starting at H.

• Y -- has the same block number as C and is often, but not necessarily equal to C (for notation C~Y see clause (6) below.)

• L -- latest, current value of this entity of the FC module (i.e. now, when looked up)

• D -- dangling, least block number of the linked chain in progress ending at H. This variable is used to record the download state eventually reaching Y (for notation D<<H see clause (6) below.)

• H -- head, sync target which typically is the value of a sync to new head request (via RPC)

The internal sync state (as opposed to the general state also including FC) is defined by the triple (C,D,H). Other parameters L and Y mentioned in (5) are considered ephemeral to the sync state. They are always used by its latest value and are not cached by the syncer.

There are two order releations and some derivatives used to describe relations beween headers or blocks.

    For blocks or headers A and B, A is said less or equal B if the      (6)
    block numbers are less or equal. Notation: A <= B.

    For blocks or headers A and B, A is said ancestor of, or equal to
    B if B is linked to A following up the lineage of parentHash fields
    of the block headers. Notation: A << B.

    The relate notation A ~ B stands for A <= B <= A which is posh for
    saying that A and B have the same block numer.

    The compact interval notation [A,B] stands for the set {X|A<<X<<B}
    and the half open interval notation stands for [A,B]-{A} (i.e. the
    interval without the left end point.)

Note that A<<B implies A<=B. Boundary conditions that hold for the clause (5) diagram are

    C ~ Y,  C in [0,L],  D in [Y,H]                                      (7)

Sync Processing

Sync starts at an idle state

    0                 H  L                                               (8)
    o-----------------o--o
    | <--- imported ---> |

where H<=L (H needs only be known by its block number.) The state parameters C and D are irrelevant here.

Following, there will be a request to advance H to a new position as indicated in the diagram below

    0            C                                                       (9)
    o------------o-------o
    | <--- imported ---> |                              D
                 Y                                      H
                 o--------------------------------------o
                 | <----------- unprocessed ----------> |

with a new sync state (C,D,H). The parameter C in clause (9) is set as the base entity of the FC module. Y is only known by its block number, Y~C. The parameter D is set to the download start position H.

The syncer then fetches the header chain (Y,H] from the network. For the syncer state (C,D,H), while iteratively fetching headers, only the parameter D will change each time a new header was fetched.

Having finished dowlnoading (Y,H] one might end up with a situation

    0             B  Z   L                                              (10)
    o-------------o--o---o
    | <--- imported ---> |
                 Y   Z                                  H
                 o---o----------------------------------o
                 | <-------------- linked ------------> |

where Z is in the intersection of [B,L]*(Y,H] with B the current base entity of the FC logic. It is only known that 0<<B<<L although in many cases B==C holds.

If there is no such Z then (Y,H] is discarded and sync processing restarts at clause (8) by resetting the sync state (e.g. to (0,0,0).)

Otherwise assume Z is the one with the largest block number of the intersection [B,L]*(Y,H]. Then the headers (Z,H] will be completed to a lineage of blocks by downloading block bodies.

    0                Z                                                  (11)
    o----------------o---o
    | <--- imported ---> |
                     Z                                  H
                     o----------------------------------o
                     | <------------ blocks ----------> |

The blocks (Z,H] will then be imported. While this happens, the internal state of the FC might change/reset so that further import becomes impossible. Even when starting import, the block Z might not be in [0,L] anymore due to some internal reset of the FC logic. In any of those cases, sync processing restarts at clause (8) by resetting the sync state.

Otherwise the block import will end up at

    0                Z                                  H   L           (12)
    o----------------o----------------------------------o---o
    | <--- imported --------------------------------------> |

with H<<L for L the current value of the latest entity of the FC module. In many cases, H==L but there are other actors running that might import blocks quickly after importing H so that H is seen as ancestor, different from L when this stage is formally done with.

Now clause (12) is equivalent to clause (8).

Running the sync process for MainNet

For syncing, a beacon node is needed that regularly informs via RPC of a recently finalised block header.

The beacon node program used here is the nimbus_beacon_node binary from the nimbus-eth2 project (any other, e.g.the light client will do.) Nimbus_beacon_node is started as

  ./run-mainnet-beacon-node.sh \
     --web3-url=http://127.0.0.1:8551 \
     --jwt-secret=/tmp/jwtsecret

where http://127.0.0.1:8551 is the URL of the sync process that receives the finalised block header (here on the same physical machine) and /tmp/jwtsecret is the shared secret file needed for mutual communication authentication.

It will take a while for nimbus_beacon_node to catch up (see the Nimbus Guide for details.)

Starting nimbus for syncing

As the syncing process is quite slow, it makes sense to pre-load the database from an Era1 archive (if available) before starting the real sync process. The command for importing an Era1 reproitory would be something like

   ./build/nimbus_execution_client import \
      --era1-dir:/path/to/main-era1/repo \
      ...

which will take its time for the full MainNet Era1 repository (but way faster than the beacon sync.)

On a system with memory considerably larger than 8GiB the nimbus binary is started on the same machine where the beacon node runs with the command

   ./build/nimbus_execution_client \
      --network=mainnet \
      --engine-api=true \
      --engine-api-port=8551 \
      --engine-api-ws=true \
      --jwt-secret=/tmp/jwtsecret \
      ...

Note that --engine-api-port=8551 and --jwt-secret=/tmp/jwtsecret match the corresponding options from the nimbus-eth2 beacon source example.

Syncing on a low memory machine

On a system with memory with 8GiB the following additional options proved useful for nimbus to reduce the memory footprint.

For the Era1 pre-load (if any) the following extra options apply to "nimbus import":

   --chunk-size=1024
   --debug-rocksdb-row-cache-size=512000
   --debug-rocksdb-block-cache-size=1500000

To start syncing, the following additional options apply to nimbus:

   --debug-beacon-chunk-size=384
   --debug-rocksdb-max-open-files=384
   --debug-rocksdb-write-buffer-size=50331648
   --debug-rocksdb-block-cache-size=1073741824
   --debug-rdb-key-cache-size=67108864
   --debug-rdb-vtx-cache-size=268435456

Also, to reduce the backlog for nimbus-eth2 stored on disk, the following changes might be considered. In the file nimbus-eth2/vendor/mainnet/metadata/config.yaml change the folloing settings

   MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024
   MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096

to

   MIN_EPOCHS_FOR_BLOCK_REQUESTS: 8
   MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 8

Caveat: These changes are not useful when running nimbus_beacon_node as a production system.

Metrics

The following metrics are defined in worker/update/metrics.nim which will be available if nimbus is compiled with the additional make flags NIMFLAGS="-d:metrics --threads:on":

Variable	Logic type	Short description
beacon_base	block height	B, increasing
beacon_latest	block height	L, increasing
beacon_coupler	block height	C, increasing
beacon_dangling	block height	D
beacon_final	block height	F, increasing
beacon_head	block height	H, increasing
beacon_target	block height	T, increasing
beacon_header_lists_staged	size	# of staged header list records
beacon_headers_unprocessed	size	# of accumulated header block numbers
beacon_block_lists_staged	size	# of staged block list records
beacon_blocks_unprocessed	size	# of accumulated body block numbers
beacon_buddies	size	# of peers working concurrently