When running the import, currently blocks are loaded in batches into a
`seq` then passed to the importer as such.
In reality, blocks are still processed one by one, so the batching does
not offer any performance advantage. It does however require that the
client wastes memory, up to several GB, on the block sequence while
they're waiting to be processed.
This PR introduces a persister that accepts these potentially large
blocks one by one and at the same time removes a number of redundant /
unnecessary copies, assignments and resets that were slowing down the
import process in general.
The current getCanonicalHead of core db should not be confused with ForkedChain.latestHeader.
Therefore we need to use getCanonicalHead to restricted case only, e.g. initializing ForkedChain.
In block processing, depending on the complexity of a transaction and
hotness of caches etc, signature checking can actually make up the
majority of time needed to process a transaction (60% observed in some
randomly sampled block ranges).
Fortunately, this is a task that trivially can be offloaded to a task
pool similar to how nimbus-eth2 does it.
This PR introduces taskpools in the most simple way possible, by
performing signature checking concurrently with other TX processing,
assigning a taskpool task per TX effectively.
With this little trick, we're in gigagas land 🎉 on my laptop!
```
INF 2024-12-10 21:05:35.170+01:00 Imported blocks
blockNumber=3874817 b... mgps=1222.707 ...
```
Tests don't use the taskpool for now because it needs manual cleanup and
we don't have a good mechanism in place. Future PR:s should address this
by creating a common shutdown sequence that also closes and cleans up
other resources like the DB.
Co-authored-by: andri lim <jangko128@gmail.com>
* `shouldPrepareTracer` always true
* simple `pop` should not copy value (reading the memory shows up in a
profiler)
* continuation code simplified
* remove some unnecessary EH
* Re-org internal descriptor `CanonicalDesc` as `PivotArc`
why:
Despite its name, `CanonicalDesc` contained a cursor arc (or leg) from
the base tree with a designated block (or Header) on its arc members
(aka blocks.) The type is used more generally than only for s block on
the canonical cursor.
Also, the `PivotArc` provides some more fields for caching intermediate
data. This simplifies managing extra arguments for some functions.
* Remove cruft
details:
No need to find cursor arc if it is given as function argument.
* Rename prototype variables `head: PivotArc` to `pvarc`
why:
Better reading
* Function and code massage, adjust names
details:
Avoid the syllable `canonical` in function names that do not strictly
apply to the canonical chain. So renaming
* findCanonicalHead() => findCursorArc()
* canonicalChain() => findHeader()
* trimCanonicalChain() => trimCursorArc()
* Combine `updateBase()` function-args into single `PivotArgs` object
why:
Will generalise action for more complex scenarios in future.
* update `calculateNewBase()` return code type => `PivotArc`
why:
So it can directly be used as argument into `updateBase()`
* Update `calculateNewBase()` for target on parent arc
* Update unit tests
* Kludge: fix `eip4844` import in `validate`
why:
Importing `validate` needs `blscurve` here or with the importing module.
* Separate out `FC` descriptor iinto separate file
why:
Needed for external descriptor access (e.g. for debugging)
* Debugging toolkit for `FC`
* Verify chain descriptor after changing state
* Cosmetics, update log and exception messages
* Update `FC` base tree updater `updateBase()`
why:
Correct `forkJunction` of canonical cursor head record. When moving
the `base`, this field would be below `base` unless updated.
* Fix `FC` chain selector `findCanonicalHead()`
why:
Given a sample ref `hash` the function searched for the unique chain
containing the block header referenced by `hash`.
Unfortunately, when searching down the ancestry lineage, the function
did not necessarily stop an the end of the sub-chain. Rather it
continued with the parent chain without noticing. So returning the
wrong result.
* When calculating new a base it must reside on cursor arc (or leg.)
why:
The finalised block argument (that will eventually be the new base)
might be moved further down the cursor arc if it is too close to the
cursor head (typically smaller than 128 blocks.)
So the finalised block selection is shifted down he cursor arc. And
it might happen that the cursor arc itself is too small and one would
end up at a parent cursor arc. This is rejected.
* Not starting a new cursor arc with a block already on another arc
why:
This leads to an inconsistent set of cursor arcs which are supposed to
be mutually disjunct.
* Tighten condition: A block that is not on the base tree must be on the DB
* One less TODO item
why:
The `base` block is ancestor to all blocks of the base tree bust stays
outside the tree.
Some fringe condition uses an opportunistic fix when the `cursor` is not in
the base tree, which is legit if `cursor != base`.
* Fixes related to Prague execution requests
Turn out the specs are changed:
- WITHDRAWAL_REQUEST_ADDRESS -> WITHDRAWAL_QUEUE_ADDRESS
- CONSOLIDATION_REQUEST_ADDRESS -> CONSOLIDATION_QUEUE_ADDRESS
- DEPOSIT_CONTRACT_ADDRESS -> only mainnet
- depositContractAddress can be configurable
Also fix bugs related to t8n tool
* Fix for evmc
* Feature: User configurable extraData when assemble a block
As evident from https://holesky.beaconcha.in/block/2657016
when nimbus-eth1 assemble a block, the extraData field is empty.
This commit will give user a chance to put his extraData or
use default value.
* Warning if extraData exceeds 32 bytes limit
* Add missing comma
* switch to Nim v2.0.12
* fix LruCache capitalization for styleCheck
* KzgProof/KzgCommitment for styleCheck
* TxEip4844 for styleCheck
* styleCheck issues in nimbus/beacon/payload_conv.nim
* ENode for styleCheck
* isOk for styleCheck
* some more styleCheck fixes
* more styleCheck fixes
---------
Co-authored-by: jangko <jangko128@gmail.com>
* prefer the spec-derived name where possible
* don't pass stateRoot to LedgerRef and friends (it doesn't do anything)
* add deprecation warning in graphql - it needs updating to use
forkedchain instead
* Update `ForkedChainRef` constructor
why:
Initialisation is based on the canonical head which is always zero
after resuming a stopped `ForkedChainRef` based import.
* Update new-base calculator
why:
There is some ambiguous code which might not do what the comment
implies. In short, an unsigned condition like `2u - 3u < 1u => false`
is coded where the comment suggests that `2 - 3 < 1 => true` is meant.
This patch fixes notorious crashes when resuming import after a stop.
* partial commit
* fixes
* remove converters too
* revert changes on nimbus_verified_proxy
* revert changes in converter
* revert changes(re-xport) in rpc_types
* update copyright year
* replace types in other binaries
* chain config bug
* fix rebase conflict imcomplete buffer
* fix more rebase buffers
* remove ditto types and converters
* fix the tests
* update copyright year
* bump nimbus-build-system to use Nim v2.0.10
* 2.0.10 fixes
* fluffy linting
* make trivial change which should trigger whole-nimbus+fluffy rebuild/ci
* Nim v2.0.10 chronicles.error/macros.error ambiguity workaround
* another contentType enum specifier
* fluffy linting
* Fix eth/common & web3 related deprecation warnings for fluffy
This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.
It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.
There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.
* Address review comments
* Remove two more unused eth/common imports
* fix: rpc can't serve blocks in db
* shift db access to forkedchainref
* cleanup
* kurtosis test fix, should fail + eth_getTransactionReceipt
* remove kurtosis not + cleanup
* alter CI check to pass
* optimize impl
* cleanup
* fix loop case
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
Jacek Sieka
andri lim
tersec
Jordan Hrycaj
Advaita Saha
Chirag Parmar
Kim De Mey