nimbus-eth1/nimbus/sync/snap
Jordan Hrycaj 4feaa2cfab
Aristo db update for short nodes key edge cases (#1887)
* Aristo: Provide key-value list signature calculator

detail:
  Simple wrappers around `Aristo` core functionality

* Update new API for `CoreDb`

details:
+ Renamed new API functions `contains()` => `hasKey()` or `hasPath()`
  which disables the `in` operator on non-boolean 	`contains()` functions
+ The functions `get()` and `fetch()` always return a not-found error if
  there is no item, available. The new functions `getOrEmpty()` and
  `mergeOrEmpty()` return an an empty `Blob` if there is no such key
  found.

* Rewrite `core_apps.nim` using new API from `CoreDb`

* Use `Aristo` functionality for calculating Merkle signatures

details:
  For debugging, the `VerifyAristoForMerkleRootCalc` can be set so
  that `Aristo` results will be verified against the legacy versions.

* Provide general interface for Merkle signing key-value tables

details:
  Export `Aristo` wrappers

* Activate `CoreDb` tests

why:
  Now, API seems to be stable enough for general tests.

* Update `toHex()` usage

why:
  Byteutils' `toHex()` is superior to `toSeq.mapIt(it.toHex(2)).join`

* Split `aristo_transcode` => `aristo_serialise` + `aristo_blobify`

why:
+ Different modules for different purposes
+ `aristo_serialise`: RLP encoding/decoding
+ `aristo_blobify`: Aristo database encoding/decoding

* Compacted representation of small nodes' links instead of Keccak hashes

why:
  Ethereum MPTs use Keccak hashes as node links if the size of an RLP
  encoded node is at least 32 bytes. Otherwise, the RLP encoded node
  value is used as a pseudo node link (rather than a hash.) Such a node
  is nor stored on key-value database. Rather the RLP encoded node value
  is stored instead of a lode link in a parent node instead. Only for
  the root hash, the top level node is always referred to by the hash.

  This feature needed an abstraction of the `HashKey` object which is now
  either a hash or a blob of length at most 31 bytes. This leaves two
  ways of representing an empty/void `HashKey` type, either as an empty
  blob of zero length, or the hash of an empty blob.

* Update `CoreDb` interface (mainly reducing logger noise)

* Fix copyright years (to make `Lint` happy)
2023-11-08 12:18:32 +00:00
..
worker Aristo db update for short nodes key edge cases (#1887) 2023-11-08 12:18:32 +00:00
README.txt Improve full sync part behaviour 4 snap sync suite (#1564) 2023-04-26 16:46:42 +01:00
constants.nim Code reorg 4 snap sync suite (#1560) 2023-04-24 21:24:07 +01:00
range_desc.nim Code reorg 4 snap sync suite (#1560) 2023-04-24 21:24:07 +01:00
worker.nim Code reorg 4 snap sync suite (#1560) 2023-04-24 21:24:07 +01:00
worker_desc.nim Unified database frontend integration (#1670) 2023-08-04 12:10:09 +01:00

README.txt

Test & debugging scenario with nimbus-eth1 client/server
========================================================


Start snap/1 server
-------------------

  # Enter nimbus directory for snap/1 protocol server.
  cd server

  # Tell nimbus to stop full sync after 2 mio blocks.
  echo 2000000 > full-limit.txt

  # Tell nimbus to use this predefined key ID
  echo 123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0 > full-id.key

  ./build/nimbus \
    --tcp-port:30319 --nat=None --sync-mode=full \
	--protocols=snap --discovery=none \
	--net-key=./full-id.key \
	--sync-ctrl-file=./full-limit.txt \
	--log-level:TRACE

  # Wait for several hours until enough blocks have been downloaded so that
  # snap sync data are available. The full 2 mio blocks are available if the
  # log ticker shows something like
  #
  # INF 2023-03-17 [..] Sync statistics (suspended) topics="full-tick" [..] persistent=#2000080 [..]
  #
  # where the persistent=#2000080 field might vary


Start snap/1 client
-------------------

  # Note: When the snap/1 server has enough blocks, the client can be started.

  # Enter nimbus directory for snap/1 protocol server
  cd client

  # Tell nimbus to use this pivot block number. This number must be smaller
  # than the 2000000 written into the file full-limit.txt above.
  echo 600000 > snap/snap-update.txt

  # Tell nimbus to stop somewhere after 1000000 blocks have been downloaded
  # with full sync follow up after snap sync has completed (2nd line of
  # external setuip file.)
  echo 1000000 >> snap/snap-update.txt

  # Tell nimbus to use this hard coded peer enode.
  echo enode://192d7e7a302bd4ff27f48d7852621e0d3cb863a6dd67dd44e0314a25a3aa866837f0d2460b4444dc66e7b7a2cd56a2de1c31b2a2ba4e23549bf3ba3b0c4f2eb5@127.0.0.1:30319 > snap/full-servers.txt

  ./build/nimbus \
    --tcp-port:30102 --nat=None --sync-mode=snap \
	--protocols=none --discovery=none \
	--static-peers-file=./full-servers.txt \
	--sync-ctrl-file=./snap-update.txt \
	--log-level:TRACE


Modifications while the programs are syncing
--------------------------------------------

  # Increasing the number in the files full/full-limit.txt or
  # snap/snap-update.txt will be recognised while running. Decreasing
  # or removing will be ignored.