status-im
/
nimbus-eth1
mirror of https://github.com/status-im/nimbus-eth1.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nimbus-eth1/nimbus/sync/snap/worker/db/hexary_error.nim

51 lines
1.2 KiB
Nim
Raw Normal View History

Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
# nimbus-eth1
# Copyright (c) 2021 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
type
HexaryDbError* = enum
NothingSerious = 0
AccountNotFound
AccountSmallerThanBase
AccountsNotSrictlyIncreasing
AccountRangesOverlap
Prep for full sync after snap (#1253) * Split fetch accounts into sub-modules details: There will be separated modules for accounts snapshot, storage snapshot, and healing for either. * Allow to rebase pivot before negotiated header why: Peers seem to have not too many snapshots available. By setting back the pivot block header slightly, the chances might be higher to find more peers to serve this pivot. Experiment on mainnet showed that setting back too much (tested with 1024), the chances to find matching snapshot peers seem to decrease. * Add accounts healing * Update variable/field naming in `worker_desc` for readability * Handle leaf nodes in accounts healing why: There is no need to fetch accounts when they had been added by the healing process. On the flip side, these accounts must be checked for storage data and the batch queue updated, accordingly. * Reorganising accounts hash ranges batch queue why: The aim is to formally cover as many accounts as possible for different pivot state root environments. Formerly, this was tried by starting the accounts batch queue at a random value for each pivot (and wrapping around.) Now, each pivot environment starts with an interval set mutually disjunct from any interval set retrieved with other pivot state roots. also: Stop fishing for more pivots in `worker` if 100% download is reached * Reorganise/update accounts healing why: Error handling was wrong and the (math. complexity of) whole process could be better managed. details: Much of the algorithm is now documented at the top of the file `heal_accounts.nim`
2022-10-08 17:20:50 +00:00
NodeNotFound
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
RlpEncoding
SlotsNotSrictlyIncreasing
Isolating sync action modules (#1249) * Miscellaneous updates TBC * Disentangled pivot2 module from snap why: Wrote as template on top of sync so it can be shared by fast and snap sync. * Renamed and relocated pivot sources * Integrated `best_pivot` module into full and snap sync why: Full sync used an older version of `best_pivot` * isolating download module from full sync why; might be shared with snap sync at a later stage
2022-09-30 08:22:14 +00:00
TrieLoopAlert
TrieIsEmpty
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
# import
Snap fetch account storage data (#1211) * Removed database write comparison statistics * Provide life storage tests data details: database dumps on external repo `nimbus-eth1`-blobs` * Update hexary tree interpolation for storage bulk tests * fetch storage update
2022-09-02 18:16:09 +00:00
DifferentNodeValueExists
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
ExpectedNodeKeyDiffers
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
Rlp2Or17ListEntries
RlpBlobExpected
RlpBranchLinkExpected
RlpExtPathEncoding
RlpNonEmptyBlobExpected
Snap sync accounts healing (#1225) * Added inspect module why: Find dangling references for trie healing support. details: + This patch set provides only the inspect module and some unit tests. + There are also extensive unit tests which need bulk data from the `nimbus-eth1-blob` module. * Alternative pivot finder why: Attempt to be faster on start up. Also tying to decouple pivot finder somehow by providing different mechanisms (this one runs in `single` mode.) * Use inspect module for healing details: + After some progress with account and storage data, the inspect facility is used to find dangling links in the database to be filled nose-wise. + This is a crude attempt to cobble together functional elements. The set up needs to be honed. * fix scheduler to avoid starting dead peers why: Some peers drop out while in `sleepAsync()`. So extra `if` clauses make sure that this event is detected early. * Bug fixes causing crashes details: + prettify.toPC(): int/intToStr() numeric range over/underflow + hexary_inspect.hexaryInspectPath(): take care of half initialised step with branch but missing index into branch array * improve handling of dropped peers in alternaive pivot finder why: Strange things may happen while querying data from the network. Additional checks make sure that the state of other peers is updated immediately. * Update trace messages * reorganise snap fetch & store schedule
2022-09-16 07:24:12 +00:00
# interpolate
AccountRepairBlocked
InternalDbInconsistency
RightBoundaryProofFailed
Snap fetch account storage data (#1211) * Removed database write comparison statistics * Provide life storage tests data details: database dumps on external repo `nimbus-eth1`-blobs` * Update hexary tree interpolation for storage bulk tests * fetch storage update
2022-09-02 18:16:09 +00:00
RootNodeMismatch
RootNodeMissing
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
# bulk storage
AddBulkItemFailed
CannotOpenRocksDbBulkSession
CommitBulkItemsFailed
NoRocksDbBackend
UnresolvedRepairNode
Snap fetch account storage data (#1211) * Removed database write comparison statistics * Provide life storage tests data details: database dumps on external repo `nimbus-eth1`-blobs` * Update hexary tree interpolation for storage bulk tests * fetch storage update
2022-09-02 18:16:09 +00:00
OSErrorException
Snap sync accounts db code reorg (#1189) * Extracted functionality into sub-modules for maintainability * Setting SST bulk load as default in `accounts_db` details: + currently, the same data are stored via rocksdb if available, and the same via embedded `storage_type` with (non-standard) prefix 200 for time comparisons + fallback to normal `put()` unless rocksdb is accessible
2022-08-15 15:51:50 +00:00
# End