Commit Graph

81 Commits

Author SHA1 Message Date
Jordan Hrycaj 89ae9621c4
Silence compiler gossip after nim upgrade (#1454)
* Silence some compiler gossip -- part 1, tx_pool

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 2, clique

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 3, misc core

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Silence some compiler gossip -- part 4, sync

details:
  Mostly removing redundant imports and `Defect` tracer after switch
  to nim 1.6

* Clique update

why:
  Missing exception annotation
2023-01-30 22:10:23 +00:00
Jordan Hrycaj 197d2b16dd
Snap sync interval range extractor (#1449)
* Update comments and test noise

* Fix boundary proofs

why:
  Where neither used in production, nor unit tested. For production, other
  methods apply to test leaf range integrity directly based of the proof
  nodes.

* Added `hexary_range()`: interval range + proof extractor

details:
+ Will be used for `snap/1` protocol handler
+ Unit tests added (also for testing left boundary proof)

todo:
  Need to verify completeness of proof nodes

* Reduce some nim 1.6 compiler noise

* Stop unit test gossip for ci tests
2023-01-30 17:50:58 +00:00
Kim De Mey a669b51ec5
Bump Nim to 1.6 and resolve the related issues (#1445)
Two unresolved items currently:
- Three tests that are temporarily disabled as they fail in the
macro_assembler code, which seems to be due to an ambigious
identifier Stop (Ops and chronos ServerCommand enum).
- i386 CI disabled as it fails at Nim compilation already. Failed
tests where already ignored for this target.
2023-01-26 13:37:19 +01:00
Jordan Hrycaj e093fa452d
Declutter snap sync unit tests (#1444)
* Extracted RocksDB timing unit tests into separate file

why:
  make space for more in main module :)

* Extracted `inspectionRunner()` unit tests into separate file

why:
  make space for more in main module :)

* Extracted `storagesRunner()` unit tests into separate file

why:
  make space for more in main module :)

* Extracted pivot checkpoint store/retrieval unit tests into separate file

why:
  make space for more in main module :)

* Extract helper functions into separate source file

* Extracted account import unit tests into separate file

why:
  make space for more in main module :)

* Rename `test_decompose()` => `test_NodeRangeDecompose()`

why:
  There will be more functions with `test_NodeRange` prefix.
2023-01-23 16:09:12 +00:00
Jordan Hrycaj 6fb48517ba
Add snap protocol service stub (#1438)
* Cosmetics, update logger `topics`

* Clean up sync/start methods in nimbus

why:
* The `protocols` list selects served (as opposed to sync) protocols only.
* The `SyncMode.Default` object is allocated with the other possible sync
  mode objects.

* Add snap service stub to `nimbus`

* Provide full set of snap response handler stubs

* Bicarb for the latest CI hiccup

why:
  Might be a change in the CI engine for MacOS.
2023-01-20 15:01:29 +00:00
Jordan Hrycaj fda7971aaf
Reorganise eth handlers (#1436)
* Reorganise eth handlers

why:
  Make space for `snap` handlers in a similar fashion.

* fix typo
2023-01-18 15:00:14 +00:00
Jordan Hrycaj 30135ab1ef
Simplify beacon stream pivot update (#1435)
* Simplify pivot update

why:
  No need to fetch the pivot header from the network when it can be
  be made available in the ivot cache

also:
  Keep `txPool` update disabled while syncing

* Cosmetics, tune down some logging noise

* Support `snap/1` without `eth/6?`

why:
  Eth is not needed here.

* Snap is an (optional) extension of `eth`

so:
  It it must be supported somehow. Nevertheless it will be currently
  unused in the snap syncer.
2023-01-18 08:31:57 +00:00
Jordan Hrycaj 707e47ac38
External beacon stream tracker (#1433)
* Register external beacon stream header

why:
  This will be used to sync the peers against.

* Update total coverage book-keeping for 100% roll-over

details:
  Provide commonly available/used function

* Replace best pivot by beacon stream tracker

details:
  Beacon stream header cache will be updated by external chain monitor via
  RPC. This cached header will then be used to sync the pivot.
2023-01-17 09:28:14 +00:00
Jordan Hrycaj a6f45e341b
Fetch-reject-reconnect loop protection (#1432)
why:
  Some peers reconnect recurrently after dialogue was found useless. The
  reconnect loop protection was in place already, albeit insufficient.

also:
  Some updates to allow setting previously constant parameters at run
  time.
2023-01-16 14:51:32 +00:00
Jordan Hrycaj 8da4002df3
Update eth/6? messages Get/PooledTransactions (#1415)
why:
  There non-existent txs must be skipped.
2023-01-13 19:55:16 +00:00
jangko 74e76e5237
remove unused 'refundGas' from evm/state_transactions 2022-12-28 01:45:56 +07:00
Jordan Hrycaj 5134bb5e04
Extract finding of missing nodes for healing into separate module (#1398)
why:
  Duplicate implementation of same functionality
2022-12-25 17:56:57 +00:00
Jordan Hrycaj 88b315bb41
Snap sync refactor healing (#1397)
* Simplify accounts healing threshold management

why:
  Was over-engineered.

details:
  Previously, healing was based on recursive hexary trie perusal.

  Due to "cheap" envelope decomposition of a range complement for the
  hexary trie, the cost of running extra laps have become time-affordable
  again and a simple trigger mechanism for healing will do.

* Control number of dangling result nodes in `hexaryInspectTrie()`

also:
+ Returns number of visited nodes available for logging so the maximum
  number of nodes can be tuned accordingly.
+ Some code and docu update

* Update names of constants

why:
  Declutter, more systematic naming

* Re-implemented `worker_desc.merge()` for storage slots

why:
  Provided as proper queue management in `storage_queue_helper`.

details:
+ Several append modes (replaces `merge()`)
+ Added third queue to record entries currently fetched by a worker. So
  another parallel running worker can safe the complete set of storage
  slots in as checkpoint. This was previously lost.

* Refactor healing

why:
  Simplify and remove deep hexary trie perusal for finding completeness.

   Due to "cheap" envelope decomposition of a range complement for the
   hexary trie, the cost of running extra laps have become time-affordable
   again and a simple trigger mechanism for healing will do.

* Docu update

* Run a storage job only once in download loop

why:
  Download failure or rejection (i.e. missing data) lead to repeated
  fetch requests until peer disconnects, otherwise.
2022-12-24 09:54:18 +00:00
Jordan Hrycaj 0f132c1d01
Snap sync fix ticker crash (#1393)
* Fix SEGFAULT showstopper

* Update logging
2022-12-20 15:38:57 +00:00
Jordan Hrycaj bd42ebb193
Snap sync refactor accounts healing (#1392)
* Relocated mothballing (i.e. swap-in preparation) logic

details:
  Mothballing was previously tested & started after downloading
  account ranges in `range_fetch_accounts`.

  Whenever current download or healing stops because of a pivot change,
  swap-in preparation is needed (otherwise some storage slots may get
  lost when swap-in takes place.)

  Also, `execSnapSyncAction()` has been moved back to `pivot_helper`.

* Reorganised source file directories

details:
  Grouped pivot focused modules into `pivot` directory

* Renamed `checkNodes`, `sickSubTries` as `nodes.check`, `nodes.missing`

why:
  Both lists are typically used together as pair. Renaming `sickSubTries`
  reflects moving away from a healing centric view towards a swap-in
  attitude.

* Multi times coverage recording

details:
  Per pivot account ranges are accumulated into coverage range set. This
  set fill eventually contain a singe range of account hashes [0..2^256]
  which amounts to 100% capacity.

  A counter has been added that is incremented whenever max capacity is
  reached. The accumulated range is then reset to empty.

  The effect of this setting is that the coverage can be evenly duplicated.
  So 200% would not accumulate on a particular region.

* Update range length comparisons (mod 2^256)

why:
  A range interval can have sizes 1..2^256 as it cannot be empty by
  definition. The number of points in a range intervals set can have
  0..2^256 points. As the scalar range is a residue class modulo 2^256,
  the residue class 0 means length 2^256 for a range interval, but can
  be 0 or 2^256 for the number of points in a range intervals set.

* Generalised `hexaryEnvelopeDecompose()`

details:
  Compile the complement of the union of some (processed) intervals and
  express this complement as a list of envelopes of sub-tries.

  This facility is directly applicable to swap-in book-keeping.

* Re-factor `swapIn()`

why:
  Good idea but baloney implementation. The main algorithm is based on
  the generalised version of `hexaryEnvelopeDecompose()` which has been
  derived from this implementation.

* Refactor `healAccounts()` using `hexaryEnvelopeDecompose()` as main driver

why:
  Previously, the hexary trie was searched recursively for dangling nodes
  which has a poor worst case performance already when the trie  is
  reasonably populated.

  The function `hexaryEnvelopeDecompose()` is a magnitude faster because
  it does not peruse existing sub-tries in order to find missing nodes
  although result is not fully compatible with the previous function.

  So recursive search is used in a limited mode only when the decomposer
  will not deliver a useful result.

* Logging & maintenance fixes

details:
  Preparation for abandoning buddy-global healing variables `node`,
  `resumeCtx`, and `lockTriePerusal`. These variable are trie-perusal
  centric which will be run on the back burner in favour of
  `hexaryEnvelopeDecompose()` which is used for accounts healing already.
2022-12-19 21:22:09 +00:00
Jordan Hrycaj d55a72ae49
Full sync peer negotiation control (#1390)
* Additional logging for scheduler

* Fix duplicate occurrence of `bestNumber`

why:
  Happened when the `block_queue` module was separated out of
  the `worker` module. Somehow testing was insufficient or skipped,
  at all.

* Update `runPool()` mixin for scheduler

details:
  Could be simplified

* Dynamically adapt pivot header negotiation mode

details:
  After accepting one peer and some timeout, do not search for more
  peers for start syncing but rather continue in relaxed mode with a
  single peer.
2022-12-18 16:06:43 +00:00
jangko eb701fd3d7
fix addKnownToPeer in wire protocol handler 2022-12-16 07:55:38 +07:00
Etan Kissling 22338b7870 bump `nim-eth` for `eip4844` support
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4844 (danksharding). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `excessDataGas` field until proper EIP4844 support exists.
https://github.com/status-im/nim-eth/pull/570
2022-12-14 11:04:13 +02:00
Jordan Hrycaj 52517d598f
Fix typo (#1364)
details:
  Accessing wrong result (out of two) leads to an exception
2022-12-13 11:13:13 +00:00
Jordan Hrycaj cc2c888a63
Snap sync swap in other pivots (#1363)
* Provide index to reconstruct missing storage slots

why;
  Pivots will be changed anymore once they are officially archived. The
  account of the archived pivots are ready to be swapped into the active
  pivot. This leaves open how to treat storage slots not fetched yet.

  Solution: when mothballing, an `account->storage-root` index is
  compiled that can be used when swapping in accounts.

* Implement swap-in from earlier pivots

details;
  When most accounts are covered by the current and previous pivot
  sessions, swapping inthe accounts and storage slots  (i.e. registering
  account ranges done) from earlier pivots takes place if there is a
  common sub-trie.

* Throttle pivot change when healing state has bean reached

why:
  There is a hope to complete the current pivot, so pivot update can be
  throttled. This is achieved by setting another minimum block number
  distance for the pivot headers. This feature is still experimental
2022-12-12 22:00:24 +00:00
Jordan Hrycaj 179b4adac3
Snap sync tweaks n fixes (#1359)
* Miscellaneous tweaks & fixes

details:
+ Catch `TransportError` exception in `legacy.nim` module
+ Fix self-calling wrapper `hexaryEnvelopeTouchedBy()`

* Update documentation, logging etc.

* Changed `checkNode` batch list `seq[Blob]` => `seq[NodeSpecs]`

why:
  The `NodeSpecs` type as used here is a tuple `(partial-path,node-key)`.

  When `checkNode` partial paths are collected, also the node key is
  available so it should be registered and not repeatedly recovered from
  the database.

* Add optional begin/end trace statement in snap scheduler

why:
  Allows to trace invoked entity and scheduler state variables
2022-12-09 13:43:55 +00:00
Jordan Hrycaj 3766eddf5a
Some updates to the envelope module (#1353)
details:
+ Add detailed error return codes
+ Remove cruft
+ Some prototype wrappers
2022-12-06 20:13:31 +00:00
Jordan Hrycaj 85de03fd6e
Rename and update dismantle => hexaryEnvelopeDecompose() (#1351)
* Rename and update dismantle => hexaryEnvelopeDecompose()

why:
+ As for naming, a positive connotation is prefered
+ The unit tests were really insufficient
+ The function result was wrong on a few boundry conditions

detail:
+ Extracted the function from `hexary_paths.nim` and re-implemented
  it together with other envelope functions => `hexary_envelope.nim`
+ Re-wrote docu for `hexaryEnvelopeDecompose()`

* Relaxed right condition for `hexaryEnvelopeDecompose()` range argument

why;
  Previously, the right point of the argument interval had to be a path
  to an allocated leaf node. While this is typically a given for accounts,
  it is easier to require an arbitrary range of paths (or keys) with
  the requirement of a `boundary proof` for left and right (i.e. enough
  nodes in the database to find the end points.)

also:
  Bug fixes for related functions (typos, missing conditions etc.)

* Add missing unit tests include file
2022-12-06 17:35:56 +00:00
jangko 4cf2ab661c
connect legacy sync to rpc/eth_syncing and graphql/syncing
fix #1333
2022-12-05 10:25:21 +07:00
jangko 56f169b23e
rename Fast Sync to Legacy Sync
fix #1332
2022-12-05 09:42:25 +07:00
jangko 94a94c5b65 implement better hardfork management 2022-12-02 13:51:42 +07:00
jangko ac2cb82a2b saner source code grouping 2022-12-02 13:51:42 +07:00
Jordan Hrycaj 44a57496d9
Snap sync interval complement method to speed up trie perusal (#1328)
* Add quick hexary trie inspector, called `dismantle()`

why:
+ Full hexary trie perusal is slow if running down leaf nodes
+ For known range of leaf nodes, work out the UInt126-complement of
  partial sub-trie paths (for existing nodes). The result should cover
  no (or only a few) sub-tries with leaf nodes.

* Extract common healing methods => `sub_tries_helper.nim`

details:
  Also apply quick hexary trie inspection tool `dismantle()`
  Replace `inspectAccountsTrie()` wrapper by `hexaryInspectTrie()`

* Re-arrange task dispatching in main peer worker

* Refactor accounts and storage slots downloaders

* Rename `HexaryDbError` => `HexaryError`
2022-11-28 09:03:23 +00:00
Etan Kissling bc3f164b97
bump `nim-eth` for `withdrawalsRoot` support (#1326)
The `BlockHeader` structure in `nim-eth` was updated with support for
EIP-4895 (withdrawals). To enable the `nim-eth` bump, the ingress of
`BlockHeader` structures has been hardened to reject headers that have
the new `withdrawalsRoot` field until proper withdrawals support exists.
https://github.com/status-im/nim-eth/pull/562
2022-11-26 15:59:19 +01:00
Jordan Hrycaj 7688148565
Snap sync can start on saved checkpoint (#1327)
* Stop negotiating pivot if peer repeatedly replies w/usesless answers

why:
  There is some fringe condition where a peer replies with legit but
  useless empty headers repetely. This goes on until somebody stops.
  We stop now.

* Rename `missingNodes` => `sickSubTries`

why:
  These (probably missing) nodes represent in reality fully or partially
  missing sub-tries. The top nodes may even exist, e.g. as a shallow
  sub-trie.

also:
  Keep track of account healing on/of by bool variable `accountsHealing`
  controlled in `pivot_helper.execSnapSyncAction()`

* Add `nimbus` option argument `snapCtx` for starting snap recovery (if any)

also:
+ Trigger the recovery (or similar) process from inside the global peer
  worker initialisation `worker.setup()` and not by the `snap.start()`
  function.
+ Have `runPool()` returned a `bool` code to indicate early stop to
  scheduler.

* Can import partial snap sync checkpoint at start

details:
 + Modified what is stored with the checkpoint in `snapdb_pivot.nim`
 + Will be loaded within `runDaemon()` if activated

* Forgot to import total coverage range

why:
  Only the top (or latest) pivot needs coverage but the total coverage
  is the list of all ranges for all pivots -- simply forgotten.
2022-11-25 14:56:42 +00:00
jangko fffe071f86
eth wire protocol: implement NewBlock and NewBlockHashes handler
and it also do invasive changes to fast sync because
they are tightly related.

fix #673
2022-11-18 01:22:31 +07:00
Jordan Hrycaj bba1bea4c8
Snap sync state save (#1302)
* Piecemeal trie inspection

details:
  Trie inspection will stop after maximum number of nodes visited.
  The inspection can be resumed using the returned state from the
  last session.

why:
  This feature allows for task switch between `piecemeal` sessions.

* Extract pivot helper code from `worker.nim` => `pivot_helper.nim`

* Accounts import will now return dangling paths from `proof` nodes

why:
  With proper bookkeeping, this can be used to start healing without
  analysing the the probably full trie.

* Update `unprocessed` account range handling

why:
  More generally, the API of a pairs of unprocessed intervals favours
  the first set and not before that is exhausted the second set comes
  into play.

  This was unfortunately implemented which caused the ranges to be
  unnecessarily fractioned. Now the number of range interval typically
  remains in the lower single digit numbers.

* Save sync state after end of downloading some accounts

details:
  restore/resume to be implemented later
2022-11-16 23:51:06 +00:00
Jordan Hrycaj 9aa925cf36
Update sync scheduler (#1297)
* Add `stop()` methods to shutdown to shutdown procedure

why:
  Nasty behaviour when hitting Ctrl-C, otherwise

* Add background service to sync scheduler

why:
  The background service will be used for sync data import and recovery
  after restart.

   It is controlled by the sync scheduler for an easy turn/on off API.

also:
  Simplified snap ticker time calc.

* Fix typo
2022-11-14 14:13:00 +00:00
jangko 43f4b99a1b
disable NewBlockHashes and NewBlock of eth wire handler after POS transition
fix #1133
2022-11-14 16:17:34 +07:00
jangko c7b3c374f0
add exception handlers to transaction exchange code 2022-11-14 12:43:05 +07:00
jangko 9dd256cae7
eth wire handlers: implement transactions exchange 2022-11-11 09:24:32 +07:00
Jordan Hrycaj 21837546c3
Fix/clarify single mode for async sync scheduler (#1292)
why:
  Single mode here means there is only such (single mode) instance
  activated but multi mode instances for other peers are allowed.

  Erroneously, multi mode instances were held back waiting while some
  single mode instance was running which reduced the number of parallel
  download peers.
2022-11-09 19:16:25 +00:00
Jordan Hrycaj e14fd4b96c
Prep for full sync after snap make 6 (#1291)
* Update log ticker, using time interval rather than ticker count

why:
  Counting and logging ticker occurrences is inherently imprecise. So
  time intervals are used.

* Use separate storage tables for snap sync data

* Left boundary proof update

why:
  Was not properly implemented, yet.

* Capture pivot in peer worker (aka buddy) tasks

why:
  The pivot environment is linked to the `buddy` descriptor. While
  there is a task switch, the pivot may change. So it is passed on as
  function argument `env` rather than retrieved from the buddy at
  the start of a sub-function.

* Split queues `fetchStorage` into `fetchStorageFull` and `fetchStoragePart`

* Remove obsolete account range returned from `GetAccountRange` message

why:
  Handler returned the wrong right value of the range. This range was
  for convenience, only.

* Prioritise storage slots if the queue becomes large

why:
  Currently, accounts processing is prioritised up until all accounts
  are downloaded. The new prioritisation has two thresholds for
  + start processing storage slots with a new worker
  + stop account processing and switch to storage processing

also:
  Provide api for `SnapTodoRanges` pair of range sets in `worker_desc.nim`

* Generalise left boundary proof for accounts or storage slots.

why:
  Detailed explanation how this works is documented with
  `snapdb_accounts.importAccounts()`.

  Instead of enforcing a left boundary proof (which is still the default),
  the importer functions return a list of `holes` (aka node paths) found in
  the argument ranges of leaf nodes. This in turn is used by the book
   keeping software for data download.

* Forgot to pass on variable in function wrapper

also:
  + Start healing not before 99% accounts covered (previously 95%)
  + Logging updated/prettified
2022-11-08 18:56:04 +00:00
Jordan Hrycaj a689e9185a
Prep for full sync after snap make 5 (#1286)
* Update docu and logging

* Extracted and updated constants from `worker_desc` into separate file

* Update and re-calibrate communication error handling

* Allow simplified pivot negotiation

why:
  This feature allows to turn off pivot negotiation so that peers agree
  on a a pivot header.

  For snap sync with fast changing pivots this only throttles the sync
  process. The finally downloaded DB snapshot is typically a merged
  version of different pivot states augmented by a healing process.

* Re-model worker queues for accounts download & healing

why:
  Currently there is only one data fetch per download or healing task.
  This task is then repeated by the scheduler after a short time. In
  many cases, this short time seems enough for some peers to decide to
  terminate connection.

* Update main task batch `runMulti()`

details:
  The function `runMulti()` is activated in quasi-parallel mode by the
  scheduler. This function calls the download, healing and fast-sync
  functions.

  While in debug mode, after each set of jobs run by this function the
  database is analysed (by the `snapdb_check` module) and the result
  printed.
2022-11-01 15:07:44 +00:00
Jordan Hrycaj a8df4c1165
Fix trie inspector for healing (#1284)
* Update logging

* Fix node hash associated with partial path for missing nodes

why:
  Healing uses the partial paths for fetching nodes from the network. The
  node hash (or key) is used to verify the node data retrieved.

  The trie inspector function returned the parent hash instead of the node hash
  with the partial path when a missing node was detected. So all nodes
  for healing were rejected.

* Must not modify sequence while looping over it
2022-10-28 08:26:17 +01:00
Jordan Hrycaj 1b4572ed3b
Prep for full sync after snap make 4 (#1282)
* Re-arrange fetching storage slots in batch module

why;
  Previously, fetching partial slot ranges first has a chance of
  terminating the worker peer 9due to network error) while there were
  many inheritable storage slots on the queue.

  Now, inheritance is checked first, then full slot ranges and finally
  partial ranges.

* Update logging

* Bundled node information for healing into single object `NodeSpecs`

why:
  Previously, partial paths and node keys were kept in separate variables.
  This approach was error prone due to copying/reassembling function
  argument objects.

  As all partial paths, keys, and node data types are more or less handled
  as `Blob`s over the network (using Eth/6x, or Snap/1) it makes sense to
  hold these `Blob`s as named field in a single object (even if not all
  fields are active for the current purpose.)

* For good housekeeping, using `NodeKey` type only for account keys

why:
  previously, a mixture of `NodeKey` and `Hash256` was used. Now, only
  state or storage root keys use the `Hash256` type.

* Always accept latest pivot (and not a slightly older one)

why;
  For testing it was tried to use a slightly older pivot state root than
  available. Some anecdotal tests seemed to suggest an advantage so that
  more peers are willing to serve on that older pivot. But this could not
  be confirmed in subsequent tests (still anecdotal, though.)

  As a side note, the distance of the latest pivot to its predecessor is
  at least 128 (or whatever the constant `minPivotBlockDistance` is
  assigned to.)

* Reshuffle name components for some file and function names

why:
  Clarifies purpose:
  "storages" becomes: "storage slots"
  "store" becomes: "range fetch"

* Stash away currently unused modules in sub-folder named "notused"
2022-10-27 14:49:28 +01:00
Jordan Hrycaj 82ceec313d
Prettify logging for snap sync environment (#1278)
* Multiple storage batches at a time

why:
  Previously only some small portion was processed at a time so the peer
  might have gone when the process was resumed at a later time

* Renamed some field of snap/1 protocol response object

why:
  Documented as `slots` is in reality a per-account list of slot lists. So
  the new name `slotLists` better reflects the nature of the beast.

* Some minor healing re-arrangements for storage slot tries

why;
  Resolving all complete inherited slots tries first in sync mode keeps
  the worker queues smaller which improves logging.

* Prettify logging, comments update etc.
2022-10-21 20:29:42 +01:00
Jordan Hrycaj c0d580715e
Remodel persistent snapdb access (#1274)
* Re-model persistent database access

why:
  Storage slots healing just run on the wrong sub-trie (i.e. the wrong
  key mapping). So get/put and bulk functions now use the definitions
  in `snapdb_desc` (earlier there were some shortcuts for `get()`.)

* Fixes: missing return code, typo, redundant imports etc.

* Remove obsolete debugging directives from `worker_desc` module

* Correct failing unit tests for storage slots trie inspection

why:
  Some pathological cases for the extended tests do not produce any
  hexary trie data. This is rightly detected by the trie inspection
  and the result checks needed to adjusted.
2022-10-20 17:59:54 +01:00
Jordan Hrycaj 096d93ab31
Remove direct support for legacy pivot finder (#1272)
why:
  Not used anymore. The current finder is good enough based on the
  the reported best header and difficulty.
2022-10-19 15:03:55 +01:00
Jordan Hrycaj 85fdb61699
Prep for full sync after snap make 3 (#1270)
* For snap sync, publish `EthWireRef` in sync descriptor

why:
  currently used for noise control

* Detect and reuse existing storage slots

* Provide healing module for storage slots

* Update statistic ticker (adding range factor for unprocessed storage)

* Complete mere function for work item ranges

why:
  Merging interval into existing partial item was missing

* Show av storage queue lengths in ticker

detail;
  Previous attempt shows average completeness which did not tell much

* Correct the meaning of the storage counter (per pivot)

detail:
  Is the # accounts that have a storage saved
2022-10-19 11:04:06 +01:00
jangko 3fa1b012e6
initial wire protocol transformation
rework on the eth wire protocol handlers.
curently still missing 4 handlers implementation.
but the framework is ready for eexpansion.
2022-10-15 19:48:21 +07:00
Jordan Hrycaj 8c7d91512b
Prep for full sync after snap mark2 (#1263)
* Rename `LeafRange` => `NodeTagRange`

* Replacing storage slot partition point by interval

why:
  The partition point only allows to describe slots `[point,high(Uint256)]`
  for fetching interval slot ranges. This has been generalised for any
  interval.

* Replacing `SnapAccountRanges` by `SnapTrieRangeBatch`

why:
  Generalised healing status for accounts, and later for storage slots.

* Improve accounts healing loop

* Split `snap_db` into accounts and storage modules

why:
  It is cleaner to have separate session descriptors for accounts and
  storage slots (based on a common base descriptor.)

  Also, persistent storage handling might be changed in future which
  requires the storage slot implementation disentangled from the accounts
  handling.

* Re-model worker queues for storage slots

why:
  There is a dynamic list of storage sub-tries, each one has to be
  treated similar to the accounts database. This applied to slot
  interval downloads as well as to healing

* Compress some return value report lists for snapdb methods

why:
  No need to report all handling details for work items that are filteres
  out and discarded, anyway.

* Remove inner loop frame from healing function

why:
  The healing function runs as a loop body already.
2022-10-14 17:40:32 +01:00
Jordan Hrycaj d53eacb854
Prep for full sync after snap (#1253)
* Split fetch accounts into sub-modules

details:
  There will be separated modules for accounts snapshot, storage snapshot,
  and healing for either.

* Allow to rebase pivot before negotiated header

why:
  Peers seem to have not too many snapshots available. By setting back the
  pivot block header slightly, the chances might be higher to find more
  peers to serve this pivot. Experiment on mainnet showed that setting back
  too much (tested with 1024), the chances to find matching snapshot peers
  seem to decrease.

* Add accounts healing

* Update variable/field naming in `worker_desc` for readability

* Handle leaf nodes in accounts healing

why:
  There is no need to fetch accounts when they had been added by the
  healing process. On the flip side, these accounts must be checked for
  storage data and the batch queue updated, accordingly.

* Reorganising accounts hash ranges batch queue

why:
  The aim is to formally cover as many accounts as possible for different
  pivot state root environments. Formerly, this was tried by starting the
  accounts batch queue at a random value for each pivot (and wrapping
  around.)

  Now, each pivot environment starts with an interval set mutually
  disjunct from any interval set retrieved with other pivot state roots.

also:
  Stop fishing for more pivots in `worker` if 100% download is reached

* Reorganise/update accounts healing

why:
  Error handling was wrong and the (math. complexity of) whole process
  could be better managed.

details:
  Much of the algorithm is now documented at the top of the file
  `heal_accounts.nim`
2022-10-08 18:20:50 +01:00
Jordan Hrycaj eca5882238
Isolating sync action modules (#1249)
* Miscellaneous updates TBC

* Disentangled pivot2 module from snap

why:
  Wrote as template on top of sync so it can be shared by fast and snap
  sync.

* Renamed and relocated pivot sources

* Integrated `best_pivot` module into full and snap sync

why:
  Full sync used an older version of `best_pivot`

* isolating download module from full sync

why;
  might be shared with snap sync at a later stage
2022-09-30 09:22:14 +01:00
jangko 86f6d284aa
initial beacon sync skeleton implementation 2022-09-17 09:08:55 +07:00