Commit Graph

3 Commits

Author SHA1 Message Date
Jordan Hrycaj 73b628491d
Clique snapshots reorg (#1169)
* Add persistent snapshot size logging

why:
  Suspecting too much space used

snapshot statistic:
  [..]
  blockNumber=2214912 nSnaps=2236 snapsTotal=1.14m
  blockNumber=2215936 nSnaps=2237 snapsTotal=1.14m
  [..]
  Persisting blocks fromBlock=2216449 toBlock=2216640
  36458496	datadir-nimbus-goerlish/data/nimbus/

* Replace legacy `lru_cache` by `keyed_queue`

why:
  `keyed_queue` generalises `lru_cache`

snapshot statistic:
  [..]
  blockNumber=2234368 nSnaps=2259 snapsTotal=1.15m
  blockNumber=2235392 nSnaps=2260 snapsTotal=1.15m
  [..]
  Persisting blocks fromBlock=2235649 toBlock=2235840
  37627288	datadir-nimbus-goerlish/data/nimbus/

* Increase persistent snapshot storage interval by 300%

snapshot statistic:
      [..]
      blockNumber=2232320 nSnaps=620 snapsTotal=0.30m
      blockNumber=2236416 nSnaps=621 snapsTotal=0.30m
      [..]
      Persisting blocks fromBlock=2237185 toBlock=2237376
      37627288	datadir-nimbus-goerlish/data/nimbus/

* Cull legacy debugging environment for clique

why:
  Chronicles provides a better choice (when properly set up)
2022-07-21 19:16:28 +01:00
Jordan Hrycaj ec9354d2d0
Jordan/poa voting header (#782)
* Provide PoA voting header generator

why:
  Handy for hive/smoke test

details:
  Header generator is a re-implementation of the generator previously
  used for the canonical reference tests.

* try fixing ci out-of-mem condition

why:
  for some reason, the ci began behaving like a real win7/i386 machine
  where gcc is limited to 64k optimiser space

* fix comments, typos ..
2021-08-03 08:15:32 +01:00
Jordan Hrycaj a0d0e35a70
Renamed source file clique_utils => clique_helpers (#762)
* Renamed source file clique_utils => clique_helpers

why:
  New name is more in line with other modules where local libraries
  are named similarly.

* re-implemented PoA verification module as clique_verify.nim

details:
  The verification code was ported from the go sources and provisionally
  stored in the clique_misc.nim source file.

todo:
  Bring it to life.

* re-design Snapshot descriptor as: ref object

why:
  Avoids some copying descriptor objects

details:
  The snapshot management in clique_snapshot.nim has been cleaned up.

todo:
  There is a lot of unnecessary copying & sub-list manipulation of
  seq[BlockHeader] lists which needs to be simplified by managing
  index intervals.

* optimised sequence handling for Clique/PoA

why:
  To much ado about nothing

details:
  * Working with shallow sequences inside PoA processing avoids
    unnecessary copying.
  * Using degenerate lists in the cliqueVerify() batch where only the
    parent (and no other ancestor) is needed.

todo:
  Expose only functions that are needed, shallow sequences should be
  handles with care.

* fix var-parameter function argument

* Activate PoA engine -- currently proof of concept

details:
  PoA engine is activated with newChain(extraValidation = true) applied
  to a PoA network.

status and todo:
  The extraValidation flag on the Chain object can be set at a later
  state which allows to pre-load parts of the block chain without
  verification. Setting it later will only go back the block chain to
  the latest epoch checkpoint. This is inherent to the Clique protocol,
  needs testing though.

  PoA engine works in fine weather mode on Goerli replay. With the
  canonical eip-225 tests, there are quite a few fringe conditions
  that fail. These can easily fudged over to make things work but need
  some more work to understand and correct properly.

* Make the last offending verification header available

why:
  Makes some fringe case tests work.

details:
  Within a failed transaction comprising several blocks, this
  feature help to identify the offending block if there was a
  PoA verification error.

* Make PoA header verifier store the final snapshot

why:
  The last snapshot needed by the verifier is the one of the parent but
  the list of authorised signer is derived from the current snapshot. So
  updating to the latest snapshot provides the latest signers list.

details:
  Also, PoA processing has been implemented as transaction in
  persistBlocks() with Clique state rollback.

  Clique tests succeed now.

* Avoiding double yields in iterator => replaced by template

why:
  Tanks to Andri who observed it (see #762)

* Calibrate logging interval and fix logging event detection

why:
  Logging interval as copied from Go implementation was too large and
  needed re-calibration. Elapsed time calculation was bonkers, negative
  the wrong way round.
2021-07-21 14:31:52 +01:00