`updateOk` is obsolete and always set to true - callers should not have
to care about this detail
also take the opportunity to clean up storage root naming
* Log/trace cancellation events in scheduler
* Provide `clear()` functions for explicitly flushing data objects
* Renaming header cache functions
why:
More systematic, all functions start with prefix `dbHeader`
* Remove `danglingParent` from layout
why:
Already provided by header cache
* Remove `couplerHash` and `headHash` from layout
why:
No need to cache, `headHash` is unused and `couplerHash` used typically
once, only.
* Remove `lastLayout` from sync descriptor
why:
No need to compare changes, saving is always triggered after actively
changing the sync layout state
* Early reject unsuitable head + finalised header from CL
why:
The finalised header is only passed by its hash so the header must be
fetched somewhere, e.g. from a peer via eth/xx.
Also, finalised headers earlier than the `base` from `FC` cannot be
handled due to the `Aristo` single state database architecture.
Luckily, on a full node, the complete block history is available so
unsuitable finalised headers are stored there already which is exploited
here to avoid unnecessary network traffic.
* Code cosmetics, remove cruft, prettify logging, remove `final` metrics
detail:
The `final` layout parameter will be deprecated and later removed
* Update/re-calibrate syncer logic documentation
why:
The current implementation sucks if the `FC` module changes the
canonical branch in the middle of completing a header chain (due
to concurrent updates by the `newPayload()` logic.)
* Implement according to re-calibrated syncer docu
details:
The implementation employs the notion of named layout states (see
`SyncLayoutState` in `worker_desc.nim`) which are derived from the
state parameter triple `(C,D,H)` as described in `README.md`.
When walking AriVtx, parsing integers and nibbles actually becomes a
hotspot - these trivial changes reduces CPU usage during initial key
cache computation by ~15%.
Currently, computed hash keys are stored in a separate column family
with respect to the MPT data they're generated from - this has several
disadvantages:
* A lot of space is wasted because the lookup key (`RootedVertexID`) is
repeated in both tables - this is 30% of the `AriKey` content!
* rocksdb must maintain in-memory bloom filters and LRU caches for said
keys, doubling its "minimal efficient cache size"
* An extra disk traversal must be made to check for existence of cached
hash key
* Doubles the amount of files on disk due to each column family being
its own set of files
Here, the two CFs are joined such that both key and data is stored in
`AriVtx`. This means:
* we save ~30% disk space on repeated lookup keys
* we save ~2gb of memory overhead that can be used to cache data instead
of indices
* we can skip storing hash keys for MPT leaf nodes - these are trivial
to compute and waste a lot of space - previously they had to present in
the `AriKey` CF to avoid having to look in two tables on the happy path.
* There is a small increase in write amplification because when a hash
value is updated for a branch node, we must write both key and branch
data - previously we would write only the key
* There's a small shift in CPU usage - instead of performing lookups in
the database, hashes for leaf nodes are (re)-computed on the fly
* We can return to slightly smaller on-disk SST files since there's
fewer of them, which should reduce disk traffic a bit
Internally, there are also other advantages:
* when clearing keys, we no longer have to store a zero hash in memory -
instead, we deduce staleness of the cached key from the presence of an
updated VertexRef - this saves ~1gb of mem overhead during import
* hash key cache becomes dedicated to branch keys since leaf keys are no
longer stored in memory, reducing churn
* key computation is a lot faster thanks to the skipped second disk
traversal - a key computation for mainnet can be completed in 11 hours
instead of ~2 days (!) thanks to better cache usage and less read
amplification - with additional improvements to the on-disk format, we
can probably get rid of the initial full traversal method of seeding the
key cache on first start after import
All in all, this PR reduces the size of a mainnet database from 160gb to
110gb and the peak memory footprint during import by ~1-2gb.
* Fixes related to Prague execution requests
Turn out the specs are changed:
- WITHDRAWAL_REQUEST_ADDRESS -> WITHDRAWAL_QUEUE_ADDRESS
- CONSOLIDATION_REQUEST_ADDRESS -> CONSOLIDATION_QUEUE_ADDRESS
- DEPOSIT_CONTRACT_ADDRESS -> only mainnet
- depositContractAddress can be configurable
Also fix bugs related to t8n tool
* Fix for evmc
* Feature: User configurable extraData when assemble a block
As evident from https://holesky.beaconcha.in/block/2657016
when nimbus-eth1 assemble a block, the extraData field is empty.
This commit will give user a chance to put his extraData or
use default value.
* Warning if extraData exceeds 32 bytes limit
* Add missing comma
* Annotate `async` functions for non-exception tracking at compile time
details:
This also requires some additional try/except catching in the function
bodies.
* Update sync logic docu to what is to be updated
why:
The understanding of details of how to accommodate for merging
sub-chains of blocks or headers have changed. Some previous set-ups
are outright wrong.
This kind of data is not used except in tests where it is used only to
create databases that don't match actual usage of aristo.
Removing simplifies future optimizations that can focus on processing
specific leaf types more efficiently.
A casualty of this removal is some test code as well as some proof
generation code that is unused - on the surface, it looks like it should
be possible to port both of these to the more specific data types -
doing so would ensure that a database written by one part of the
codebase can interact with the other - as it stands, there is confusion
on this point since using the proof generation code will result in a
database of a shape that is incompatible with the rest of eth1.
* Clear rejected sync target so that it would not be processed again
* Use in-memory table to stash headers after FCU import has started
why:
After block imported has started, there is no way to save/stash block
headers persistently. The FCU handlers always maintain a positive
transaction level and in some instances the current transaction is
flushed and re-opened.
This patch fixes an exception thrown when a block header has gone
missing.
* When resuming sync, delete stale headers and state
why:
Deleting headers saves some persistent space that would get lost
otherwise. Deleting the state after resuming prevents from race
conditions.
* On clean start hibernate sync `deamon` entity before first update from CL
details:
Only reduces services are running
* accept FCU from CL
* fetch finalised header after accepting FCY (provides hash only)
* Improve text/meaning of some log messages
* Revisit error handling for useless peers
why:
A peer is abandoned from if the error score is too high. This was not
properly handled for some fringe case when the error was detected at
staging time but fetching via eth/xx was ok.
* Clarify `break` meaning by using labelled `break` statements
* Fix action how to commit when sync target has been reached
why:
The sync target block number might precede than latest FCU block number.
This happens when the engine API squeezes in some request to execute
and import subsequent blocks.
This patch fixes and assert thrown when after reaching target the latest
FCU block number is higher than the expected target block number.
* Update TODO list
* switch to Nim v2.0.12
* fix LruCache capitalization for styleCheck
* KzgProof/KzgCommitment for styleCheck
* TxEip4844 for styleCheck
* styleCheck issues in nimbus/beacon/payload_conv.nim
* ENode for styleCheck
* isOk for styleCheck
* some more styleCheck fixes
* more styleCheck fixes
---------
Co-authored-by: jangko <jangko128@gmail.com>
* Clarifying/commenting FCU setup condition & small fixes, comments etc.
* Update some logging
* Reorg metrics updater and activation
* Better `async` responsiveness
why:
Block import does not allow `async` task activation while
executing. So allow potential switch after each imported
block (rather than a group of 32 blocks.)
* Handle resuming after previous sync followed by import
why:
In this case the ledger state is more recent than the saved
sync state. So this is considered a pristine sync where any
previous sync state is forgotten.
This fixes some assert thrown because of inconsistent internal
state at some point.
* Provide option for clearing saved beacon sync state before starting syncer
why:
It would resume with the last state otherwise which might be undesired
sometimes.
Without RPC available, the syncer typically stops and terminates with
the canonical head larger than the base/finalised head. The latter one
will be saved as database/ledger state and the canonical head as syncer
target. Resuming syncing here will repeat itself.
So clearing the syncer state can prevent from starting the syncer
unnecessarily avoiding useless actions.
* Allow workers to request syncer shutdown from within
why:
In one-trick-pony mode (after resuming without RPC support) the
syncer can be stopped from within soavoiding unnecessary polling.
In that case, the syncer can (theoretically) be restarted externally
with `startSync()`.
* Terminate beacon sync after a single run target is reached
why:
Stops doing useless polling (typically when there is no RPC available)
* Remove crufty comments
* Tighten state reload condition when resuming
why:
Some pathological case might apply if the syncer is stopped while the
distance between finalised block and head is very large and the FCU
base becomes larger than the locked finalised state.
* Verify that finalised number from CL is at least FCU base number
why:
The FCU base number is determined by the database, non zero if
manually imported. The finalised number is passed via RPC by the CL
node and will increase over time. Unless fully synced, this number
will be pretty low.
On the other hand, the FCU call `forkChoice()` will eventually fail
if the `finalizedHash` argument refers to something outside the
internal chain starting at the FCU base block.
* Remove support for completing interrupted sync without RPC support
why:
Simplifies start/stop logic
* Rmove unused import
* prefer the spec-derived name where possible
* don't pass stateRoot to LedgerRef and friends (it doesn't do anything)
* add deprecation warning in graphql - it needs updating to use
forkedchain instead
When `nimbus import` runs, we end up with a database without MPT roots
leading to long startup times the first time one is needed.
Computing the state root is slow because the on-disk order based on
VertexID sorting does not match the trie traversal order and therefore
makes lookups inefficent.
Here we introduce a helper that speeds up this computation by traversing
the trie in on-disk order and computing the trie hashes bottom up
instead - even though this leads to some redundant reads of nodes that
we cannot yet compute, it's still a net win as leaves and "bottom"
branches make up the majority of the database.
This PR also addresses a few other sources of inefficiency largely due
to the separation of AriKey and AriVtx into their own column families.
Each column family is its own LSM tree that produces hundreds of SST
filtes - with a limit of 512 open files, rocksdb must keep closing and
opening files which leads to expensive metadata reads during random
access.
When rocksdb makes a lookup, it has to read several layers of files for
each lookup. Ribbon filters to skip over files that don't have the
requested data but when these filters are not in memory, reading them is
slow - this happens in two cases: when opening a file and when the
filter has been evicted from the LRU cache. Addressing the open file
limit solves one source of inefficiency, but we must also increase the
block cache size to deal with this problem.
* rocksdb.max_open_files increased to 2048
* per-file size limits increased so that fewer files are created
* WAL size increased to avoid partial flushes which lead to small files
* rocksdb block cache increased
All these increases of course lead to increased memory usage, but at
least performance is acceptable - in the future, we'll need to explore
options such as joining AriVtx and AriKey and/or reducing the row count
(by grouping branch layers under a single vertexid).
With this PR, the mainnet state root can be computed in ~8 hours (down
from 2-3 days) - not great, but still better.
Further, we write all keys to the database, also those that are less
than 32 bytes - because the mpt path is part of the input, it is very
rare that we actually hit a key like this (about 200k such entries on
mainnet), so the code complexity is not worth the benefit really, in the
current database layout / design.
* remove redundant abstraction
* fix misleading raises - the implementation actually swallows errors or
panics (depending on how many other layers of abstraction we penetrate
before detecting it)
* blocks can be bigger than the default 1mb when json-rpc-encoded - this
happens on sepolia for example
* json-rpc bump improves debug logging and fixes a number of bugs
* json-serialization bump fixes a crash on invalid arrays in json data
At some point, it would probably be better to compute the maximum block
size from actual block constraints, though this is somewhat tricky and
depends on gas limits etc. Until then, 16mb should be plenty.
With this, sepolia can be synced :)
* Update comments & logs
* Do not start beacon sync unless there is possibly something to do
why:
It would continue polling without having any effect other than
logging. Now it will not start unless there is RPC available
or there was a previously interrupted sync to be resumed.
* Accept finalised hash from RPC with the canon header as well
* Reorg internal sync descriptor(s)
details:
Update target from RPC to provide the `consensus header` as well as
the `finalised` block number
why:
Prepare for using `importBlock()` instead of `persistBlocks()`
* Cosmetic updates
details:
+ Collect all pretty printers in `helpers.nim`
+ Remove unused return codes from function prototype
* Use `importBlock()` + `forkChoice()` rather than `persistBlocks()`
* Update logging and metrics
* Update docu
* Update `ForkedChainRef` constructor
why:
Initialisation is based on the canonical head which is always zero
after resuming a stopped `ForkedChainRef` based import.
* Update new-base calculator
why:
There is some ambiguous code which might not do what the comment
implies. In short, an unsigned condition like `2u - 3u < 1u => false`
is coded where the comment suggests that `2 - 3 < 1 => true` is meant.
This patch fixes notorious crashes when resuming import after a stop.
* partial commit
* fixes
* remove converters too
* revert changes on nimbus_verified_proxy
* revert changes in converter
* revert changes(re-xport) in rpc_types
* update copyright year
* replace types in other binaries
* chain config bug
* fix rebase conflict imcomplete buffer
* fix more rebase buffers
* remove ditto types and converters
* fix the tests
* update copyright year
* rename nimbus binary to nimbus_execution_client
* additional replacements
* makefile and dockerfile
* fix ci building errors
* github workflows
* improved Makefile target
---------
Co-authored-by: Pedro Miranda <pedro.miranda@nimbus.team>
* Fix fringe condition clarifying how to handle an empty range
why:
The `interval_set` module would treat an undefined interval construct
`[2,1]` as`[2,2]` (the right bound being `max(2,1)`.)
* Use the `consensus head` rather than the `finalised` block as sync target
why:
The former is ahead of the `finalised` block.
* In ctx descriptor rename `final` field to `target`
* Update docu, rename `F` -> `T`
* bump nimbus-build-system to use Nim v2.0.10
* 2.0.10 fixes
* fluffy linting
* make trivial change which should trigger whole-nimbus+fluffy rebuild/ci
* Nim v2.0.10 chronicles.error/macros.error ambiguity workaround
* another contentType enum specifier
* fluffy linting
* Fix eth/common & web3 related deprecation warnings for fluffy
This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.
It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.
There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.
* Address review comments
* Remove two more unused eth/common imports
* fix: rpc can't serve blocks in db
* shift db access to forkedchainref
* cleanup
* kurtosis test fix, should fail + eth_getTransactionReceipt
* remove kurtosis not + cleanup
* alter CI check to pass
* optimize impl
* cleanup
* fix loop case
* Rename `base` -> `coupler`, `B` -> `C`
why:
Glossary: The jargon `base` is used for the `base state` block number
which can be smaller than what is now the `coupler`.
* Rename `global state` -> `base`, `T` -> `B`
why:
See glossary
* Rename `final` -> `end`, `F` -> `E`
why:
See glossary. Previously, `final` denoted some finalised block but not
`the finalised` block from the glossary (which is maximal.)
* Properly name finalised block as such, rename `Z` -> `F`
why:
See glossary
* Rename `least` -> `dangling`, `L` -> `D`
* Metrics update (variables not covered yet)
* Docu update and corrections
* Logger updates
* Remove obsolete `skeleton*Key` kvt columns from `storage_types` module
* Remove `--sync-mode` option from nimbus config
why:
Currently there is only one sync mode available.
* Rename `flare` -> `beacon`, but not base module folder and nim source
why:
The name `flare` was used do designate an alternative `beacon` mode that.
Leaving the base folder and source as-is for a moment, makes it easier
to read change diffs.
* Rename `flare` base module folder and nim source: `flare` -> `beacon`
* Dissolve legacy `sync/types.nim` into `*/eth/eth_types.nim`
* Flare sync: Simplify scheduler and remove `runSingle()` method
why:
`runSingle()` is not used anymore (main purpose was for negotiating
best headers in legacy full sync.)
Also, `runMulti()` was renamed `runPeer()`
* Flare sync: Move `chain` field from `sync_desc` -> `worker_desc`
* Flare sync: Remove handler descriptor lateral reference
why:
Not used anymore. It enabled to turn on/off eth handler activity with
regards to the sync state, i.e.from with in the sync worker.
* Flare sync: Update `Hash256` and other deprecated `std/eth` symbols
* Protocols: Update `Hash256` and other deprecated `std/eth` symbols
* Eth handler: Update `Hash256` and other deprecated `std/eth` symbols
* Update flare TODO
* Remove redundant `sync/type` import
why:
The import module `type` has been removed
* Remove duplicate implementation
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
* Cosmetics, small fixes, add stashed headers verifier
* Remove direct `Era1` support
why:
Era1 is indirectly supported by using the import tool before syncing.
* Clarify database persistent save function.
why:
Function relied on the last saved state block number which was wrong.
It now relies on the tx-level. If it is 0, then data are saved directly.
Otherwise the task that owns the tx will do it.
* Extracted configuration constants into separate file
* Enable single peer mode for debugging
* Fix peer losing issue in multi-mode
details:
Running concurrent download peers was previously programmed as running
a batch downloading and storing ~8k headers and then leaving the `async`
function to be restarted by a scheduler.
This was unfortunate because of occasionally occurring long waiting
times for restart.
While the time gap until restarting were typically observed a few
millisecs, there were always a few outliers which well exceed several
seconds. This seemed to let remote peers run into timeouts.
* Prefix function names `unprocXxx()` and `stagedYyy()` by `headers`
why:
There will be other `unproc` and `staged` modules.
* Remove cruft, update logging
* Fix accounting issue
details:
When staging after fetching headers from the network, there was an off
by 1 error occurring when the result was by one smaller than requested.
Also, a whole range was mis-accounted when a peer was terminating
connection immediately after responding.
* Fix slow/error header accounting when fetching
why:
Originally set for detecting slow headers in a row, the counter
was wrongly extended to general errors.
* Ban peers for a while that respond with too few headers continuously
why:
Some peers only returned one header at a time. If these peers sit on a
farm, they might collectively slow down the download process.
* Update RPC beacon header updater
why:
Old function hook has slightly changed its meaning since it was used
for snap sync. Also, the old hook is used by other functions already.
* Limit number of peers or set to single peer mode
details:
Merge several concepts, single peer mode being one of it.
* Some code clean up, fixings for removing of compiler warnings
* De-noise header fetch related sources
why:
Header download looks relatively stable, so general debugging is not
needed, anymore. This is the equivalent of removing the scaffold from
the part of the building where work has completed.
* More clean up and code prettification for headers stuff
* Implement body fetch and block import
details:
Available headers are used stage blocks by combining existing headers
with newly fetched blocks. Then these blocks are imported/executed via
`persistBlocks()`.
* Logger cosmetics and cleanup
* Remove staged block queue debugging
details:
Feature still available, just not executed anymore
* Docu, logging update
* Update/simplify `runDaemon()`
* Re-calibrate block body requests and soft config for import blocks batch
why:
* For fetching, larger fetch requests are mostly truncated anyway on
MainNet.
* For executing, smaller batch sizes reduce the memory needed for the
price of longer execution times.
* Update metrics counters
* Docu update
* Some fixes, formatting updates, etc.
* Update `borrowed` type: uint -. uint64
also:
Always convert to `uint64` rather than `uint` where appropriate
* ForkedChainRef.forkchoice: Skip newBase calculation and skip chain finalization if finalizedHash is zero
* Fix ForkedChainRef.forkChoice: do nothing if headHash is the same with cursorHash
* Fix stupid bug in engine API FCU when calling ForkedChainRef.forkChoice
* Wire RPC server API to nimbus RPC manager
* Add test case
* Use default(Hash256) in ForkedChainRef
* init style for Hash256
https://github.com/status-im/nim-eth/pull/733 updates `Hash256` to
become an array instead of an object - unfortunately, nim does not allow
constructing arrays with `name()`, so this PR changes it to `default`
which works with both.
* lint
* fix: nimbus state ahead of era history
* comments
* fix: suggestions
* fix: messages
* fix edge case resume
* check from last file
* formatting
* fix: typo
* fix: unwanted quit before rlp import
* batch database key writes during `computeKey` calls
* log progress when there are many keys to update
* avoid evicting the vertex cache when traversing the trie for key
computation purposes
* avoid storing trivial leaf hashes that directly can be loaded from the
vertex
* Add missing leaf cache update when a leaf turns to a branch with two
leaves (on merge) and vice versa (on delete) - this could lead to stale
leaves being returned from the cache causing validation failures - it
didn't happen because the leaf caches were not being used efficiently :)
* Replace `seq` with `ArrayBuf` in `Hike` allowing it to become
allocation-free - this PR also works around an inefficiency in nim in
returning large types via a `var` parameter
* Use the leaf cache instead of `getVtxRc` to fetch recent leaves - this
makes the vertex cache more efficient at caching branches because fewer
leaf requests pass through it.
The storage leaf cache was being circumvented when actually fetching
leaves and was instead only being filled with items :/
Also avoids an expensive copy when fetching account data (broadly,
variant objects are comparatively expensive to copy and fetching
accounts is a hotspot)
* move pfx out of variant which avoids pointless field type panic checks
and copies on access
* make `VertexRef` a non-inheritable object which reduces its memory
footprint and simplifies its use - it's also unclear from a semantic
point of view why inheritance makes sense for storing keys
Compared to `keyed_queue`, `minilru` uses significantly less memory, in
particular for the 32-byte hash keys where `kq` stores several copies of
the key redundantly.
Jacek Sieka
Advaita Saha
andri lim
Jordan Hrycaj
tersec
Chirag Parmar
Pedro Miranda
Kim De Mey