status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/sync/snap/worker/db/hexary_error.nim

76 lines
1.6 KiB
Nim
Raw Normal View History

Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
# nimbus-eth1
# Copyright (c) 2021 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
type
Snap sync interval complement method to speed up trie perusal (#1328) * Add quick hexary trie inspector, called `dismantle()` why: + Full hexary trie perusal is slow if running down leaf nodes + For known range of leaf nodes, work out the UInt126-complement of partial sub-trie paths (for existing nodes). The result should cover no (or only a few) sub-tries with leaf nodes. * Extract common healing methods => `sub_tries_helper.nim` details: Also apply quick hexary trie inspection tool `dismantle()` Replace `inspectAccountsTrie()` wrapper by `hexaryInspectTrie()` * Re-arrange task dispatching in main peer worker * Refactor accounts and storage slots downloaders * Rename `HexaryDbError` => `HexaryError`
2022-11-28 09:03:23 +00:00
HexaryError* = enum
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
NothingSerious = 0
AccountNotFound
AccountsNotSrictlyIncreasing
AccountRangesOverlap
Prep for full sync after snap make 6 (#1291) * Update log ticker, using time interval rather than ticker count why: Counting and logging ticker occurrences is inherently imprecise. So time intervals are used. * Use separate storage tables for snap sync data * Left boundary proof update why: Was not properly implemented, yet. * Capture pivot in peer worker (aka buddy) tasks why: The pivot environment is linked to the `buddy` descriptor. While there is a task switch, the pivot may change. So it is passed on as function argument `env` rather than retrieved from the buddy at the start of a sub-function. * Split queues `fetchStorage` into `fetchStorageFull` and `fetchStoragePart` * Remove obsolete account range returned from `GetAccountRange` message why: Handler returned the wrong right value of the range. This range was for convenience, only. * Prioritise storage slots if the queue becomes large why: Currently, accounts processing is prioritised up until all accounts are downloaded. The new prioritisation has two thresholds for + start processing storage slots with a new worker + stop account processing and switch to storage processing also: Provide api for `SnapTodoRanges` pair of range sets in `worker_desc.nim` * Generalise left boundary proof for accounts or storage slots. why: Detailed explanation how this works is documented with `snapdb_accounts.importAccounts()`. Instead of enforcing a left boundary proof (which is still the default), the importer functions return a list of `holes` (aka node paths) found in the argument ranges of leaf nodes. This in turn is used by the book keeping software for data download. * Forgot to pass on variable in function wrapper also: + Start healing not before 99% accounts covered (previously 95%) + Logging updated/prettified
2022-11-08 18:56:04 +00:00
LowerBoundAfterFirstEntry
LowerBoundProofError
Prep for full sync after snap (#1253) * Split fetch accounts into sub-modules details: There will be separated modules for accounts snapshot, storage snapshot, and healing for either. * Allow to rebase pivot before negotiated header why: Peers seem to have not too many snapshots available. By setting back the pivot block header slightly, the chances might be higher to find more peers to serve this pivot. Experiment on mainnet showed that setting back too much (tested with 1024), the chances to find matching snapshot peers seem to decrease. * Add accounts healing * Update variable/field naming in `worker_desc` for readability * Handle leaf nodes in accounts healing why: There is no need to fetch accounts when they had been added by the healing process. On the flip side, these accounts must be checked for storage data and the batch queue updated, accordingly. * Reorganising accounts hash ranges batch queue why: The aim is to formally cover as many accounts as possible for different pivot state root environments. Formerly, this was tried by starting the accounts batch queue at a random value for each pivot (and wrapping around.) Now, each pivot environment starts with an interval set mutually disjunct from any interval set retrieved with other pivot state roots. also: Stop fishing for more pivots in `worker` if 100% download is reached * Reorganise/update accounts healing why: Error handling was wrong and the (math. complexity of) whole process could be better managed. details: Much of the algorithm is now documented at the top of the file `heal_accounts.nim`
2022-10-08 17:20:50 +00:00
NodeNotFound
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
RlpEncoding
SlotsNotSrictlyIncreasing
Isolating sync action modules (#1249) * Miscellaneous updates TBC * Disentangled pivot2 module from snap why: Wrote as template on top of sync so it can be shared by fast and snap sync. * Renamed and relocated pivot sources * Integrated `best_pivot` module into full and snap sync why: Full sync used an older version of `best_pivot` * isolating download module from full sync why; might be shared with snap sync at a later stage
2022-09-30 08:22:14 +00:00
TrieLoopAlert
TrieIsEmpty
Snap sync interval complement method to speed up trie perusal (#1328) * Add quick hexary trie inspector, called `dismantle()` why: + Full hexary trie perusal is slow if running down leaf nodes + For known range of leaf nodes, work out the UInt126-complement of partial sub-trie paths (for existing nodes). The result should cover no (or only a few) sub-tries with leaf nodes. * Extract common healing methods => `sub_tries_helper.nim` details: Also apply quick hexary trie inspection tool `dismantle()` Replace `inspectAccountsTrie()` wrapper by `hexaryInspectTrie()` * Re-arrange task dispatching in main peer worker * Refactor accounts and storage slots downloaders * Rename `HexaryDbError` => `HexaryError`
2022-11-28 09:03:23 +00:00
TrieIsLockedForPerusal
Snap sync can start on saved checkpoint (#1327) * Stop negotiating pivot if peer repeatedly replies w/usesless answers why: There is some fringe condition where a peer replies with legit but useless empty headers repetely. This goes on until somebody stops. We stop now. * Rename `missingNodes` => `sickSubTries` why: These (probably missing) nodes represent in reality fully or partially missing sub-tries. The top nodes may even exist, e.g. as a shallow sub-trie. also: Keep track of account healing on/of by bool variable `accountsHealing` controlled in `pivot_helper.execSnapSyncAction()` * Add `nimbus` option argument `snapCtx` for starting snap recovery (if any) also: + Trigger the recovery (or similar) process from inside the global peer worker initialisation `worker.setup()` and not by the `snap.start()` function. + Have `runPool()` returned a `bool` code to indicate early stop to scheduler. * Can import partial snap sync checkpoint at start details: + Modified what is stored with the checkpoint in `snapdb_pivot.nim` + Will be loaded within `runDaemon()` if activated * Forgot to import total coverage range why: Only the top (or latest) pivot needs coverage but the total coverage is the list of all ranges for all pivots -- simply forgotten.
2022-11-25 14:56:42 +00:00
TooManyProcessedChunks
Snap sync state save (#1302) * Piecemeal trie inspection details: Trie inspection will stop after maximum number of nodes visited. The inspection can be resumed using the returned state from the last session. why: This feature allows for task switch between `piecemeal` sessions. * Extract pivot helper code from `worker.nim` => `pivot_helper.nim` * Accounts import will now return dangling paths from `proof` nodes why: With proper bookkeeping, this can be used to start healing without analysing the the probably full trie. * Update `unprocessed` account range handling why: More generally, the API of a pairs of unprocessed intervals favours the first set and not before that is exhausted the second set comes into play. This was unfortunately implemented which caused the ranges to be unnecessarily fractioned. Now the number of range interval typically remains in the lower single digit numbers. * Save sync state after end of downloading some accounts details: restore/resume to be implemented later
2022-11-16 23:51:06 +00:00
TooManySlotAccounts
External beacon stream tracker (#1433) * Register external beacon stream header why: This will be used to sync the peers against. * Update total coverage book-keeping for 100% roll-over details: Provide commonly available/used function * Replace best pivot by beacon stream tracker details: Beacon stream header cache will be updated by external chain monitor via RPC. This cached header will then be used to sync the pivot.
2023-01-17 09:28:14 +00:00
NoAccountsYet
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
Snap sync interval range extractor (#1449) * Update comments and test noise * Fix boundary proofs why: Where neither used in production, nor unit tested. For production, other methods apply to test leaf range integrity directly based of the proof nodes. * Added `hexary_range()`: interval range + proof extractor details: + Will be used for `snap/1` protocol handler + Unit tests added (also for testing left boundary proof) todo: Need to verify completeness of proof nodes * Reduce some nim 1.6 compiler noise * Stop unit test gossip for ci tests
2023-01-30 17:50:58 +00:00
# range
LeafNodeExpected
FailedNextNode
Rename and update dismantle => hexaryEnvelopeDecompose() (#1351) * Rename and update dismantle => hexaryEnvelopeDecompose() why: + As for naming, a positive connotation is prefered + The unit tests were really insufficient + The function result was wrong on a few boundry conditions detail: + Extracted the function from `hexary_paths.nim` and re-implemented it together with other envelope functions => `hexary_envelope.nim` + Re-wrote docu for `hexaryEnvelopeDecompose()` * Relaxed right condition for `hexaryEnvelopeDecompose()` range argument why; Previously, the right point of the argument interval had to be a path to an allocated leaf node. While this is typically a given for accounts, it is easier to require an arbitrary range of paths (or keys) with the requirement of a `boundary proof` for left and right (i.e. enough nodes in the database to find the end points.) also: Bug fixes for related functions (typos, missing conditions etc.) * Add missing unit tests include file
2022-12-06 17:35:56 +00:00
# nearby/boundary proofs
NearbyExtensionError
NearbyBranchError
NearbyGarbledNode
NearbyNestingTooDeep
NearbyUnexpectedNode
NearbyFailed
NearbyEmptyPath
NearbyLeafExpected
NearbyDanglingLink
Some updates to the envelope module (#1353) details: + Add detailed error return codes + Remove cruft + Some prototype wrappers
2022-12-06 20:13:31 +00:00
# envelope
DecomposeDegenerated
Snap sync refactor accounts healing (#1392) * Relocated mothballing (i.e. swap-in preparation) logic details: Mothballing was previously tested & started after downloading account ranges in `range_fetch_accounts`. Whenever current download or healing stops because of a pivot change, swap-in preparation is needed (otherwise some storage slots may get lost when swap-in takes place.) Also, `execSnapSyncAction()` has been moved back to `pivot_helper`. * Reorganised source file directories details: Grouped pivot focused modules into `pivot` directory * Renamed `checkNodes`, `sickSubTries` as `nodes.check`, `nodes.missing` why: Both lists are typically used together as pair. Renaming `sickSubTries` reflects moving away from a healing centric view towards a swap-in attitude. * Multi times coverage recording details: Per pivot account ranges are accumulated into coverage range set. This set fill eventually contain a singe range of account hashes [0..2^256] which amounts to 100% capacity. A counter has been added that is incremented whenever max capacity is reached. The accumulated range is then reset to empty. The effect of this setting is that the coverage can be evenly duplicated. So 200% would not accumulate on a particular region. * Update range length comparisons (mod 2^256) why: A range interval can have sizes 1..2^256 as it cannot be empty by definition. The number of points in a range intervals set can have 0..2^256 points. As the scalar range is a residue class modulo 2^256, the residue class 0 means length 2^256 for a range interval, but can be 0 or 2^256 for the number of points in a range intervals set. * Generalised `hexaryEnvelopeDecompose()` details: Compile the complement of the union of some (processed) intervals and express this complement as a list of envelopes of sub-tries. This facility is directly applicable to swap-in book-keeping. * Re-factor `swapIn()` why: Good idea but baloney implementation. The main algorithm is based on the generalised version of `hexaryEnvelopeDecompose()` which has been derived from this implementation. * Refactor `healAccounts()` using `hexaryEnvelopeDecompose()` as main driver why: Previously, the hexary trie was searched recursively for dangling nodes which has a poor worst case performance already when the trie is reasonably populated. The function `hexaryEnvelopeDecompose()` is a magnitude faster because it does not peruse existing sub-tries in order to find missing nodes although result is not fully compatible with the previous function. So recursive search is used in a limited mode only when the decomposer will not deliver a useful result. * Logging & maintenance fixes details: Preparation for abandoning buddy-global healing variables `node`, `resumeCtx`, and `lockTriePerusal`. These variable are trie-perusal centric which will be run on the back burner in favour of `hexaryEnvelopeDecompose()` which is used for accounts healing already.
2022-12-19 21:22:09 +00:00
DecomposeDisjunct
Some updates to the envelope module (#1353) details: + Add detailed error return codes + Remove cruft + Some prototype wrappers
2022-12-06 20:13:31 +00:00
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
# import
Snap fetch account storage data (#1211) * Removed database write comparison statistics * Provide life storage tests data details: database dumps on external repo `nimbus-eth1`-blobs` * Update hexary tree interpolation for storage bulk tests * fetch storage update
2022-09-02 18:16:09 +00:00
DifferentNodeValueExists
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
ExpectedNodeKeyDiffers
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
Rlp2Or17ListEntries
RlpBlobExpected
RlpBranchLinkExpected
RlpExtPathEncoding
RlpNonEmptyBlobExpected
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
# interpolate
AccountRepairBlocked
InternalDbInconsistency
RightBoundaryProofFailed
Snap fetch account storage data (#1211) * Removed database write comparison statistics * Provide life storage tests data details: database dumps on external repo `nimbus-eth1`-blobs` * Update hexary tree interpolation for storage bulk tests * fetch storage update
2022-09-02 18:16:09 +00:00
RootNodeMismatch
RootNodeMissing
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
# bulk storage
AddBulkItemFailed
CannotOpenRocksDbBulkSession
CommitBulkItemsFailed
NoRocksDbBackend
UnresolvedRepairNode
Snap fetch account storage data (#1211) * Removed database write comparison statistics * Provide life storage tests data details: database dumps on external repo `nimbus-eth1`-blobs` * Update hexary tree interpolation for storage bulk tests * fetch storage update
2022-09-02 18:16:09 +00:00
OSErrorException
Snap sync state save (#1302) * Piecemeal trie inspection details: Trie inspection will stop after maximum number of nodes visited. The inspection can be resumed using the returned state from the last session. why: This feature allows for task switch between `piecemeal` sessions. * Extract pivot helper code from `worker.nim` => `pivot_helper.nim` * Accounts import will now return dangling paths from `proof` nodes why: With proper bookkeeping, this can be used to start healing without analysing the the probably full trie. * Update `unprocessed` account range handling why: More generally, the API of a pairs of unprocessed intervals favours the first set and not before that is exhausted the second set comes into play. This was unfortunately implemented which caused the ranges to be unnecessarily fractioned. Now the number of range interval typically remains in the lower single digit numbers. * Save sync state after end of downloading some accounts details: restore/resume to be implemented later
2022-11-16 23:51:06 +00:00
StateRootNotFound
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
# End