nimbus-eth1/nimbus/sync/snap/worker.nim

97 lines
2.9 KiB
Nim
Raw Normal View History

# Nimbus
# Copyright (c) 2021 Status Research & Development GmbH
# Licensed under either of
# * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or
# http://www.apache.org/licenses/LICENSE-2.0)
# * MIT license ([LICENSE-MIT](LICENSE-MIT) or
# http://opensource.org/licenses/MIT)
# at your option. This file may not be copied, modified, or distributed
# except according to those terms.
## Sync mode pass multiplexer
## ==========================
##
## Pass state diagram:
## ::
## <init> -> <snap-sync> -> <full-sync> ---+
## ^ |
## | |
## +----------+
##
{.push raises: [].}
import
chronicles,
chronos,
./range_desc,
./worker/pass,
./worker_desc
Snap sync simplify object inheritance (#1098) * Reorg SnapPeerBase descriptor, notably start/stop flags details: Instead of using three boolean flags startedFetch, stopped, and stopThisState a single enum type is used with values SyncRunningOk, SyncStopRequest, and SyncStopped. * Restricting snap to eth66 and later why: Id-tracked request/response wire protocol can handle overlapped responses when requests are sent in row. * Align function names with source code file names why: Easier to reconcile when following the implemented logic. * Update trace logging (want file locations) why: The macros previously used hid the relevant file location (when `chroniclesLineNumbers` turned on.) It rather printed the file location of the template that was wrapping `trace`. * Use KeyedQueue table instead of sequence why: Quick access, easy configuration as LRU or FIFO with max entries (currently LRU.) * Dissolve `SnapPeerEx` object extension into `SnapPeer` why; It is logically cleaner and more obvious not to inherit from `SnapPeerBase` but to specify opaque field object references of the merged `SnapPeer` object. These can then be locally inherited. * Dissolve `SnapSyncEx` object extension into `SnapSync` why; It is logically cleaner and more obvious not to inherit from `SnapSyncEx` but to specify opaque field object references of the `SnapPeer` object. These can then be locally inherited. Also, in the re-factored code here the interface descriptor `SnapSyncCtx` inherited `SnapSyncEx` which was sub-optimal (OO inheritance makes it easier to work with call back functions.)
2022-05-23 16:53:19 +00:00
logScope:
topics = "snap-worker"
Prep for full sync after snap (#1253) * Split fetch accounts into sub-modules details: There will be separated modules for accounts snapshot, storage snapshot, and healing for either. * Allow to rebase pivot before negotiated header why: Peers seem to have not too many snapshots available. By setting back the pivot block header slightly, the chances might be higher to find more peers to serve this pivot. Experiment on mainnet showed that setting back too much (tested with 1024), the chances to find matching snapshot peers seem to decrease. * Add accounts healing * Update variable/field naming in `worker_desc` for readability * Handle leaf nodes in accounts healing why: There is no need to fetch accounts when they had been added by the healing process. On the flip side, these accounts must be checked for storage data and the batch queue updated, accordingly. * Reorganising accounts hash ranges batch queue why: The aim is to formally cover as many accounts as possible for different pivot state root environments. Formerly, this was tried by starting the accounts batch queue at a random value for each pivot (and wrapping around.) Now, each pivot environment starts with an interval set mutually disjunct from any interval set retrieved with other pivot state roots. also: Stop fishing for more pivots in `worker` if 100% download is reached * Reorganise/update accounts healing why: Error handling was wrong and the (math. complexity of) whole process could be better managed. details: Much of the algorithm is now documented at the top of the file `heal_accounts.nim`
2022-10-08 17:20:50 +00:00
# ------------------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------------------
template ignoreException(info: static[string]; code: untyped) =
try:
code
except CatchableError as e:
error "Exception at " & info & ":", name=($e.name), msg=(e.msg)
# ------------------------------------------------------------------------------
# Public start/stop and admin functions
# ------------------------------------------------------------------------------
proc setup*(ctx: SnapCtxRef): bool =
## Global set up
ctx.passInitSetup()
ignoreException("setup"):
ctx.passActor.setup(ctx)
true
proc release*(ctx: SnapCtxRef) =
## Global clean up
ignoreException("release"):
ctx.passActor.release(ctx)
ctx.passInitRelease()
proc start*(buddy: SnapBuddyRef): bool =
## Initialise worker peer
ignoreException("start"):
result = buddy.ctx.passActor.start(buddy)
proc stop*(buddy: SnapBuddyRef) =
## Clean up this peer
ignoreException("stop"):
buddy.ctx.passActor.stop(buddy)
# ------------------------------------------------------------------------------
# Public functions, sync handler multiplexers
# ------------------------------------------------------------------------------
proc runDaemon*(ctx: SnapCtxRef) {.async.} =
## Sync processsing multiplexer
ignoreException("runDaemon"):
await ctx.passActor.daemon(ctx)
proc runSingle*(buddy: SnapBuddyRef) {.async.} =
## Sync processsing multiplexer
ignoreException("runSingle"):
await buddy.ctx.passActor.single(buddy)
proc runPool*(buddy: SnapBuddyRef, last: bool; laps: int): bool =
## Sync processsing multiplexer
ignoreException("runPool"):
result = buddy.ctx.passActor.pool(buddy,last,laps)
proc runMulti*(buddy: SnapBuddyRef) {.async.} =
## Sync processsing multiplexer
ignoreException("runMulti"):
await buddy.ctx.passActor.multi(buddy)
Prep for full sync after snap make 6 (#1291) * Update log ticker, using time interval rather than ticker count why: Counting and logging ticker occurrences is inherently imprecise. So time intervals are used. * Use separate storage tables for snap sync data * Left boundary proof update why: Was not properly implemented, yet. * Capture pivot in peer worker (aka buddy) tasks why: The pivot environment is linked to the `buddy` descriptor. While there is a task switch, the pivot may change. So it is passed on as function argument `env` rather than retrieved from the buddy at the start of a sub-function. * Split queues `fetchStorage` into `fetchStorageFull` and `fetchStoragePart` * Remove obsolete account range returned from `GetAccountRange` message why: Handler returned the wrong right value of the range. This range was for convenience, only. * Prioritise storage slots if the queue becomes large why: Currently, accounts processing is prioritised up until all accounts are downloaded. The new prioritisation has two thresholds for + start processing storage slots with a new worker + stop account processing and switch to storage processing also: Provide api for `SnapTodoRanges` pair of range sets in `worker_desc.nim` * Generalise left boundary proof for accounts or storage slots. why: Detailed explanation how this works is documented with `snapdb_accounts.importAccounts()`. Instead of enforcing a left boundary proof (which is still the default), the importer functions return a list of `holes` (aka node paths) found in the argument ranges of leaf nodes. This in turn is used by the book keeping software for data download. * Forgot to pass on variable in function wrapper also: + Start healing not before 99% accounts covered (previously 95%) + Logging updated/prettified
2022-11-08 18:56:04 +00:00
# ------------------------------------------------------------------------------
# End
# ------------------------------------------------------------------------------