nimbus-eth1/nimbus/sync/flare/README.md

Syncing
=======

Syncing blocks is performed in two partially overlapping phases

* loading the header chains into separate database tables
* removing headers from the headers chain, fetching the rest of the
  block the header belongs to and executing it

Header chains
-------------

The header chains are the triple of

* a consecutively linked chain of headers starting starting at Genesis
* followed by a sequence of missing headers
* followed by a consecutively linked chain of headers ending up at a
  finalised block header received from the consensus layer

A sequence *@[h(1),h(2),..]* of block headers is called a consecutively
linked chain if

* block numbers join without gaps, i.e. *h(n).number+1 == h(n+1).number*
* parent hashes match, i.e. *h(n).hash == h(n+1).parentHash*

General header chains layout diagram

      G                B                     L                F              (1)
      o----------------o---------------------o----------------o--->
      | <-- linked --> | <-- unprocessed --> | <-- linked --> |

Here, the single upper letter symbols *G*, *B*, *L*, *F* denote block numbers.
For convenience, these letters are also identified with its associated block
header or the full block. Saying *"the header G"* is short for *"the header
with block number G"*.

Meaning of *G*, *B*, *L*, *F*:

* *G* -- Genesis block number *#0*
* *B* -- base, maximal block number of linked chain starting at *G*
* *L* -- least, minimal block number of linked chain ending at *F* with *B <= L*
* *F* -- final, some finalised block

This definition implies *G <= B <= L <= F* and the header chains can uniquely
be described by the triple of block numbers *(B,L,F)*.

### Storage of header chains:

Some block numbers from the set *{w|G<=w<=B}* may correspond to finalised
blocks which may be stored anywhere. If some block numbers do not correspond
to finalised blocks, then the headers must reside in the *flareHeader*
database table. Of course, due to being finalised such block numbers constitute
a sub-chain starting at *G*.

The block numbers from the set *{w|L<=w<=F}* must reside in the *flareHeader*
database table. They do not correspond to finalised blocks.

### Header chains initialisation:

Minimal layout on a pristine system

      G                                                                      (2)
      B
      L
      F
      o--->

When first initialised, the header chains are set to *(G,G,G)*.

### Updating header chains:

A header chain with an non empty open interval *(B,L)* can be updated only by
increasing *B* or decreasing *L* by adding headers so that the linked chain
condition is not violated.

Only when the open interval *(B,L)* vanishes the right end *F* can be increased
by *Z* say. Then

* *B==L* beacuse interval *(B,L)* is empty
* *B==F* because *B* is maximal

and the header chains *(F,F,F)* (depicted in *(3)*) can be set to *(B,Z,Z)*
(as depicted in *(4)*.)

Layout before updating of *F*

                       B                                                     (3)
                       L
      G                F                     Z
      o----------------o---------------------o---->
      | <-- linked --> |

New layout with *Z*

                                             L'                              (4)
      G                B                     F'
      o----------------o---------------------o---->
      | <-- linked --> | <-- unprocessed --> |

with *L'=Z* and *F'=Z*.

Note that diagram *(3)* is a generalisation of *(2)*.


### Complete header chain:

The header chain is *relatively complete* if it satisfies clause *(3)* above
for *G < B*. It is *fully complete* if *F==Z*. It should be obvious that the
latter condition is temporary only on a live system (as *Z* is permanently
updated.)

If a *relatively complete* header chain is reached for the first time, the
execution layer can start running an importer in the background compiling
or executing blocks (starting from block number *#1*.) So the ledger database
state will be updated incrementally.

Imported block chain
--------------------

The following imported block chain diagram amends the layout *(1)*:

      G                  T       B                     L                F    (5)
      o------------------o-------o---------------------o----------------o-->
      | <-- imported --> |       |                     |                |
      | <-------  linked ------> | <-- unprocessed --> | <-- linked --> |


where *T* is the number of the last imported and executed block. Coincidentally,
*T* also refers to the global state of the ledger database.

The headers corresponding to the half open interval `(T,B]` can be completed by
fetching block bodies and then imported/executed.

Running the sync process for *MainNet*
--------------------------------------

For syncing, a beacon node is needed that regularly informs via *RPC* of a
recently finalised block header.

The beacon node program used here is the *nimbus_beacon_node* binary from the
*nimbus-eth2* project (any other will do.) *Nimbus_beacon_node* is started as

      ./run-mainnet-beacon-node.sh \
         --web3-url=http://127.0.0.1:8551 \
         --jwt-secret=/tmp/jwtsecret

where *http://127.0.0.1:8551* is the URL of the sync process that receives the
finalised block header (here on the same physical machine) and `/tmp/jwtsecret`
is the shared secret file needed for mutual communication authentication.

It will take a while for *nimbus_beacon_node* to catch up (see the
[Nimbus Guide](https://nimbus.guide/quick-start.html) for details.)

### Starting `nimbus` for syncing

As the sync process is quite slow, it makes sense to pre-load the database
with data from an `Era1` archive (if available) before starting the real
sync process. The command would be something like

       ./build/nimbus import \
          --era1-dir:/path/to/main-era1/repo \
          ...

which will take a while for the full *MainNet* era1 repository (but way faster
than the sync.)

On a system with memory considerably larger than *8GiB* the *nimbus*
binary is started on the same machine where the beacon node runs as

       ./build/nimbus \
          --network=mainnet \
          --sync-mode=flare \
          --engine-api=true \
          --engine-api-port=8551 \
          --engine-api-ws=true \
          --jwt-secret=/tmp/jwtsecret \
          ...

Note that *--engine-api-port=8551* and *--jwt-secret=/tmp/jwtsecret* match
the corresponding options from the *nimbus-eth2* beacon source example.

### Syncing on a low memory machine

On a system with memory with *8GiB* the following additional options proved
useful for *nimbus* to reduce the memory footprint.

For the *Era1* pre-load (if any) the following extra options apply to
"*nimbus import*":

       --chunk-size=1024
       --debug-rocksdb-row-cache-size=512000
       --debug-rocksdb-block-cache-size=1500000

To start syncing, the following additional options apply to *nimbus*:

       --debug-flare-chunk-size=384
       --debug-rocksdb-max-open-files=384
       --debug-rocksdb-write-buffer-size=50331648
       --debug-rocksdb-block-cache-size=1073741824
       --debug-rdb-key-cache-size=67108864
       --debug-rdb-vtx-cache-size=268435456

Also, to reduce the backlog for *nimbus-eth2* stored on disk, the following
changes might be considered. For file
*nimbus-eth2/vendor/mainnet/metadata/config.yaml* change setting constants:

       MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024
       MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096
to

       MIN_EPOCHS_FOR_BLOCK_REQUESTS: 8
       MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 8

Caveat: These changes are not useful when running *nimbus_beacon_node* as a
production system.

Metrics
-------

The following metrics are defined in *worker/update/metrics.nim* which will
be available if *nimbus* is compiled with the additional make flags
*NIMFLAGS="-d:metrics \-\-threads:on"*:

| *Variable*                    | *Logic type* | *Short description* |
|:------------------------------|:------------:|:--------------------|
|                               |              |                     |
| flare_state_block_number      | block height | **T**, *increasing* |
| flare_base_block_number       | block height | **B**, *increasing* |
| flare_least_block_number      | block height | **L**               |
| flare_final_block_number      | block height | **F**, *increasing* |
| flare_beacon_block_number     | block height | **Z**, *increasing* |
|                               |              |                     |
| flare_headers_staged_queue_len| size | # of staged header list records      |
| flare_headers_unprocessed     | size | # of accumulated header block numbers|
| flare_blocks_staged_queue_len | size | # of staged block list records       |
| flare_blocks_unprocessed      | size | # of accumulated body block numbers  |
|                               |              |                     |
| flare_number_of_buddies       | size         | # of working peers  |