nimbus-eth1/nimbus/sync/flare/README.md

Syncing
=======

Syncing blocks is performed in two partially overlapping phases

* loading the header chains into separate database tables
* removing headers from the headers chain, fetching the rest of the
  block the header belongs to and executing it

Header chains
-------------

The header chains are the triple of

* a consecutively linked chain of headers starting starting at Genesis
* followed by a sequence of missing headers
* followed by a consecutively linked chain of headers ending up at a
  finalised block header received from the consensus layer

A sequence *@[h(1),h(2),..]* of block headers is called a consecutively
linked chain if

* block numbers join without gaps, i.e. *h(n).number+1 == h(n+1).number*
* parent hashes match, i.e. *h(n).hash == h(n+1).parentHash*

General header chains layout diagram

      G                B                     L                F              (1)
      o----------------o---------------------o----------------o--->
      | <-- linked --> | <-- unprocessed --> | <-- linked --> |

Here, the single upper letter symbols *G*, *B*, *L*, *F* denote block numbers.
For convenience, these letters are also identified with its associated block
header or the full block. Saying *"the header G"* is short for *"the header
with block number G"*.

Meaning of *G*, *B*, *L*, *F*:

* *G* -- Genesis block number *#0*
* *B* -- base, maximal block number of linked chain starting at *G*
* *L* -- least, minimal block number of linked chain ending at *F* with *B <= L*
* *F* -- final, some finalised block

This definition implies *G <= B <= L <= F* and the header chains can uniquely
be described by the triple of block numbers *(B,L,F)*.

Storage of header chains:
-------------------------

Some block numbers from the set *{w|G<=w<=B}* may correspond to finalised
blocks which may be stored anywhere. If some block numbers do not correspond
to finalised blocks, then the headers must reside in the *flareHeader*
database table. Of course, due to being finalised such block numbers constitute
a sub-chain starting at *G*.

The block numbers from the set *{w|L<=w<=F}* must reside in the *flareHeader*
database table. They do not correspond to finalised blocks.

Header chains initialisation:
-----------------------------

Minimal layout on a pristine system

      G                                                                      (2)
      B
      L
      F
      o--->

When first initialised, the header chains are set to *(G,G,G)*.

Updating header chains:
-----------------------

A header chain with an non empty open interval *(B,L)* can be updated only by
increasing *B* or decreasing *L* by adding headers so that the linked chain
condition is not violated.

Only when the open interval *(B,L)* vanishes the right end *F* can be increased
by *Z* say. Then

* *B==L* beacuse interval *(B,L)* is empty
* *B==F* because *B* is maximal

and the header chains *(F,F,F)* (depicted in *(3)*) can be set to *(B,Z,Z)*
(as depicted in *(4)*.)

Layout before updating of *F*

                       B                                                     (3)
                       L
      G                F                     Z
      o----------------o---------------------o---->
      | <-- linked --> |

New layout with *Z*

                                             L'                              (4)
      G                B                     F'
      o----------------o---------------------o---->
      | <-- linked --> | <-- unprocessed --> |

with *L'=Z* and *F'=Z*.

Note that diagram *(3)* is a generalisation of *(2)*.


Era1 repository support:
------------------------

For the initial blocks, *Era1* repositories are supported for *MainNet*
and *Sepolia*. They might be truncated at the top, condition is that provide
consecutive blocks (like the headr chains) and start at *Genesis*.

In that case, the position *B* will be immediately set to the largest available
*Era1* block number without storing any headers.

Complete header chain:
----------------------

The header chain is *relatively complete* if it satisfies clause *(3)* above
for *G < B*. It is *fully complete* if *F==Z*. It should be obvious that the
latter condition is temporary only on a live system (as *Z* is permanently
updated.)

If a *relatively complete* header chain is reached for the first time, the
execution layer can start running an importer in the background compiling
or executing blocks (starting from block number *#1*.) So the ledger database
state will be updated incrementally.