nimbus-eth1/nimbus/sync/snap/worker/db
Jordan Hrycaj bd42ebb193
Snap sync refactor accounts healing (#1392)
* Relocated mothballing (i.e. swap-in preparation) logic

details:
  Mothballing was previously tested & started after downloading
  account ranges in `range_fetch_accounts`.

  Whenever current download or healing stops because of a pivot change,
  swap-in preparation is needed (otherwise some storage slots may get
  lost when swap-in takes place.)

  Also, `execSnapSyncAction()` has been moved back to `pivot_helper`.

* Reorganised source file directories

details:
  Grouped pivot focused modules into `pivot` directory

* Renamed `checkNodes`, `sickSubTries` as `nodes.check`, `nodes.missing`

why:
  Both lists are typically used together as pair. Renaming `sickSubTries`
  reflects moving away from a healing centric view towards a swap-in
  attitude.

* Multi times coverage recording

details:
  Per pivot account ranges are accumulated into coverage range set. This
  set fill eventually contain a singe range of account hashes [0..2^256]
  which amounts to 100% capacity.

  A counter has been added that is incremented whenever max capacity is
  reached. The accumulated range is then reset to empty.

  The effect of this setting is that the coverage can be evenly duplicated.
  So 200% would not accumulate on a particular region.

* Update range length comparisons (mod 2^256)

why:
  A range interval can have sizes 1..2^256 as it cannot be empty by
  definition. The number of points in a range intervals set can have
  0..2^256 points. As the scalar range is a residue class modulo 2^256,
  the residue class 0 means length 2^256 for a range interval, but can
  be 0 or 2^256 for the number of points in a range intervals set.

* Generalised `hexaryEnvelopeDecompose()`

details:
  Compile the complement of the union of some (processed) intervals and
  express this complement as a list of envelopes of sub-tries.

  This facility is directly applicable to swap-in book-keeping.

* Re-factor `swapIn()`

why:
  Good idea but baloney implementation. The main algorithm is based on
  the generalised version of `hexaryEnvelopeDecompose()` which has been
  derived from this implementation.

* Refactor `healAccounts()` using `hexaryEnvelopeDecompose()` as main driver

why:
  Previously, the hexary trie was searched recursively for dangling nodes
  which has a poor worst case performance already when the trie  is
  reasonably populated.

  The function `hexaryEnvelopeDecompose()` is a magnitude faster because
  it does not peruse existing sub-tries in order to find missing nodes
  although result is not fully compatible with the previous function.

  So recursive search is used in a limited mode only when the decomposer
  will not deliver a useful result.

* Logging & maintenance fixes

details:
  Preparation for abandoning buddy-global healing variables `node`,
  `resumeCtx`, and `lockTriePerusal`. These variable are trie-perusal
  centric which will be run on the back burner in favour of
  `hexaryEnvelopeDecompose()` which is used for accounts healing already.
2022-12-19 21:22:09 +00:00
..
notused Snap sync refactor accounts healing (#1392) 2022-12-19 21:22:09 +00:00
hexary_desc.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00
hexary_envelope.nim Snap sync refactor accounts healing (#1392) 2022-12-19 21:22:09 +00:00
hexary_error.nim Snap sync refactor accounts healing (#1392) 2022-12-19 21:22:09 +00:00
hexary_import.nim Prep for full sync after snap make 4 (#1282) 2022-10-27 14:49:28 +01:00
hexary_inspect.nim Snap sync tweaks n fixes (#1359) 2022-12-09 13:43:55 +00:00
hexary_interpolate.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00
hexary_nearby.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00
hexary_paths.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00
rocky_bulk_load.nim Snap sync accounts db code reorg (#1189) 2022-08-15 16:51:50 +01:00
snapdb_accounts.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00
snapdb_desc.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00
snapdb_persistent.nim Snap sync interval complement method to speed up trie perusal (#1328) 2022-11-28 09:03:23 +00:00
snapdb_pivot.nim implement better hardfork management 2022-12-02 13:51:42 +07:00
snapdb_storage_slots.nim Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) 2022-12-06 17:35:56 +00:00