Commit Graph

65 Commits

Author SHA1 Message Date
Jacek Sieka c210885b73
eth: bump to new types (#2660)
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
2024-09-29 14:37:09 +02:00
andri lim 178d77ab31
Implement EIP-7002 and EIP-7251 (#2616) 2024-09-12 16:09:46 +07:00
andri lim 6503d51b44
Implement EIP-6110: Execution layer triggered deposits (#2612)
* Implement EIP-6110: Execution layer triggered deposits

* Implement EIP-6110 of t8n tool

* Avoid unnecessary DepositRequestType check

* Avoid using 'result' in t8n helpers

* Fix logs collection and deposits validation
2024-09-12 16:09:46 +07:00
andri lim 38c58c4feb
Implement EIP-2935: Serve historical block hashes from state (#2606)
* Implement EIP-2935: Serve historical block hashes from state

* Fix EIP-2935 in t8n
2024-09-10 09:52:03 +00:00
Jacek Sieka 226cdb7c68
avoid exceptions, tx copy (#2569)
* avoid some exceptions using the new compile-time strformat
* avoid copying the full transaction only to normalise 2 fields
(expensive)
2024-08-19 09:42:07 +02:00
Jacek Sieka bdc86b3fd4
small cleanups (#2526)
* remove some redundant EH
* avoid pessimising move (introduces a copy in this case!)
* shift less data around when reading era files (reduces stack usage)
2024-07-26 12:32:01 +07:00
andri lim 916f88a373
Use block number or timestamp to determine fork rules (#2496)
* Use block number or timestamp to determine fork rules

Avoid confusion raised by `forkGTE` usage where block informations are present.

* Get rid of forkGTE
2024-07-17 17:05:53 +07:00
Jacek Sieka 72947b3647
odds and ends (#2481)
small cleanups to reduce memory allocations
2024-07-13 20:42:49 +02:00
andri lim f04f30c72b
Reduce EVM complexity by removing forkOverride (#2448)
* Reduce EVM complexity by removing forkOverride

* Fixes
2024-07-04 15:48:36 +02:00
andri lim cd21c4fbec
ForkedChain implementation (#2405)
* ForkedChain implementation

- revamp test_blockchain_json using ForkedChain
- re-enable previously failing test cases.

* Remove excess error handling

* Avoid reloading parent header

* Do not force base update

* Write baggage to database

* Add findActiveChain to finalizedSegment

* Create new stagingTx in addBlock

* Check last stateRoot existence in test_blockchain_json

* Resolve rebase conflict

* More precise nomenclature for block import cursor

* Ensure bad block nor imported and good block not rejected

* finalizeSegment become forkChoice and align with engine API forkChoice spec

* Display reason when good block rejected

* Fix comments

* Put BaseDistance into CalculateNewBase equation

* Separate finalizedHash from baseHash

* Add more doAssert constraint

* Add push raises: []
2024-06-26 07:27:48 +07:00
Jacek Sieka f294d1e086
Clear account cache after each block (#2411)
When processing long ranges of blocks, the account cache grows unbounded
which cause huge memory spikes.

Here, we move the cache to a second-level cache after each block - the
second-level cache is cleared on the next block after that which creates
a simple LRU effect.

There's a small performance cost of course, though overall the freed-up
memory can now be reassigned to the rocksdb row cache which not only
makes up for the loss but overall leads to a performance increase.

The bump to 2gb of rocksdb row cache here needs more testing but is
slightly less and loosely basedy on the savings from this PR and the
circular ref fix in #2408 - another way to phrase this is that it's
better to give rocksdb more breathing room than let the memory sit
unused until circular ref collection happens ;)
2024-06-25 07:30:32 +02:00
andri lim 99ff8dc876
Fix t8n: blobGasUsed exceeds allowance issue (#2407)
* Fix t8n: blobGasUsed exceeds allowance issue

* Put blobGasUsed validation into transaction precessing pipeline
2024-06-24 07:56:24 +02:00
Jacek Sieka 9cf7e6aea3
Avoid creating database transaction for every block (#2379)
Broadly, when importing blocks we don't need a transaction / frame per
block because we can simply abort the whole update and try again with a
smaller range if we find a faulty block.

Of course, this applies mainly to semi-trusted blocks where we're not
expected to fail in applying them - this could be blocks either from
files or header-verified blocks as given by consensus.
2024-06-17 15:28:44 +02:00
andri lim 61a809cf4d
Remove EVM indirect imports and unused EVM errors (#2370)
Those indirect imports are used when there was two EVMs.
2024-06-17 09:56:39 +02:00
Jacek Sieka 242bbf03fc
Light verification and storage mode for import (#2367)
When performing block import, we can batch state root verifications and
header checks, doing them only once per chunk of blocks, assuming that
the other blocks in the batch are valid by extension.

When we're not generating receipts, we can also skip per-transaction
state root computation pre-byzantium, which is what provides a ~20%
speedup in this PR, at least on those early blocks :)

We also stop storing transactions, receipts and uncles redundantly when
importing from era1 - there is no need to waste database storage on this
when we can load it from the era1 file (eventually).
2024-06-15 11:22:37 +02:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Jacek Sieka c48b527eea
simplify error handling in block processing (#2337)
* ValidationResult -> Result
* get rid of mixed exception / other styles
2024-06-11 17:50:22 +02:00
Jacek Sieka 0b32078c4b
Consolidate block type for block processing (#2325)
This PR consolidates the split header-body sequences into a single EthBlock
sequence and cleans up the fallout from that which significantly reduces
block processing overhead during import thanks to less garbage collection
and fewer copies of things all around.

Notably, since the number of headers must always match the number of bodies,
we also get rid of a pointless degree of freedom that in the future could
introduce unnecessary bugs.

* only read header and body from era file
* avoid several unnecessary copies along the block processing way
* simplify signatures, cleaning up unused arguemnts and returns
* use `stew/assign2` in a few strategic places where the generated
  nim assignent is slow and add a few `move` to work around poor
  analysis in nim 1.6 (will need to be revisited for 2.0)

```
stats-20240607_2223-a814aa0b.csv vs stats-20240608_0714-21c1d0a9.csv
                       bps_x     bps_y     tps_x        tps_y    bpsd    tpsd    timed
block_number
(498305, 713245]    1,540.52  1,809.73  2,361.58  2775.340189  17.63%  17.63%  -14.92%
(713245, 928185]      730.36    865.26  1,715.90  2028.973852  18.01%  18.01%  -15.21%
(928185, 1143126]     663.03    789.10  2,529.26  3032.490771  19.79%  19.79%  -16.28%
(1143126, 1358066]    393.46    508.05  2,152.50  2777.578119  29.13%  29.13%  -22.50%
(1358066, 1573007]    370.88    440.72  2,351.31  2791.896052  18.81%  18.81%  -15.80%
(1573007, 1787947]    283.65    335.11  2,068.93  2441.373402  17.60%  17.60%  -14.91%
(1787947, 2002888]    287.29    342.11  2,078.39  2474.179448  18.99%  18.99%  -15.91%
(2002888, 2217828]    293.38    343.16  2,208.83   2584.77457  17.16%  17.16%  -14.61%
(2217828, 2432769]    140.09    167.86  1,081.87  1296.336926  18.82%  18.82%  -15.80%

blocks: 1934464, baseline: 3h13m1s, contender: 2h43m47s
bpsd (mean): 19.55%
tpsd (mean): 19.55%
Time (total): -29m13s, -15.14%
```
2024-06-09 16:32:20 +02:00
web3-developer db8c5b90bd
Cleanup stateless and block witness code. (#2295)
* Cleanup unneeded stateless and block witness code. Keeping MultiKeys which is used in the eth_getProofsByBlockNumber RPC endpoint which is needed for the Fluffy state network bridge.

* Rename generateWitness flag to collectWitnessData to better describe what the flag does. We only collect the keys of the touched accounts and storage slots but no block witness generation is supported for now.

* Move remaining stateless code into nimbus directory.

* Add vmstate parameter to ChainRef to fix test.

* Exclude *.in from check copyright year

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-06-08 15:05:00 +07:00
andri lim b3a5c67532
Remove exceptions from EVM (#2314)
* Remove exception from evm memory

* Remove exception from gas meter

* Remove exception from stack

* Remove exception from precompiles

* Remove exception from gas_costs

* Remove exception from op handlers

* Remove exception from op dispatcher

* Remove exception from call_evm

* Remove exception from EVM

* Fix tools and tests

* Remove exception from EVMC

* fix evmc

* Fix evmc

* Remove remnants of async evm stuff

* Remove superflous error handling

* Proc to func

* Fix errors detected by CI

* Fix EVM op call stack usage

* REmove exception handling from getVmState

* Better error message instead of just doAssert

* Remove unused validation

* Remove superflous catchRaise

* Use results.expect instead of unsafeValue
2024-06-07 15:24:32 +07:00
Jordan Hrycaj e9eae4df70
Core db disable legacy api n remove distinct tries (#2299)
* CoreDb: Remove crufty second/off-site KVT

why:
  Was used to allow late `Clique` to store directly to disk

* CoreDb: Remove prune flag related functionality

why:
  Is completely legacy stuff

* CoreDb: Remove dependence on legacy API (tests unsupported yet)

why:
  Does not fully support Aristo

* Re-factoring `state_db` using new API

details:
  Only minimum changes needed to compile `nimbus`

* Update tests and aux modules

* Turn off legacy API and remove `distinct_tries`

comment:
  The legacy API has now cruft status, will be removed soon

* Fix copyright years

* Update rpc for verified proxy

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-05 20:52:04 +00:00
Jacek Sieka 7f76586214
Speed up account ledger a little (#2279)
`persist` is a hotspot when processing blocks because it is run at least
once per transaction and loops over the entire account cache every time.

Here, we introduce an extra `dirty` map that keeps track of all accounts
that need checking during `persist` which fixes the immediate
inefficiency, though probably this could benefit from a more thorough
review - we also get rid of the unused clearCache flag - we start with
a fresh cache on every fresh vmState.

* avoid unnecessary code hash comparisons
* avoid unnecessary copies when iterating
* use EMPTY_CODE_HASH throughout for code hash comparison
2024-06-02 21:21:29 +02:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
tersec e895c0baeb
rm Clique consensus method support and Goerli network (#2219)
* rm Clique consensus method support and Goerli network

* rm a few more SealingEngineRef and GoerliNets
2024-05-25 16:12:14 +02:00
Jordan Hrycaj 2c322054e0
Attempt to roll back stateless mode implementation in a single PR (#2209)
* Attempt to roll back stateless mode implementation in a single PR

why:
+ Stateless mode is not fully working and in the way
+ Single PR should make it feasible to investigate for a possible
  re-implementation

* Fix copyright year

* Fix annotation for exception (evmc mode)
2024-05-22 21:01:19 +00:00
web3-developer 48630ccb03
Experimental RPC endpoints for generating block witnesses (#1977)
* Completed draft implementation of witness JSON-RPC endpoints for portal network bridge.

* Updated Nimbus RPC configuration to support enabling experimental endpoints.

* Moved witness verification tests.

* Added json test for getProof.

* Added main procs to new tests to fix test suite.

* Added getBlockWitness test to blockchain json test suite.

* Added tests for experimental RPC endpoints and improved the API to support returning state proofs from before or after block execution.

* Correctly rollback transaction in getBlockWitness proc.
2024-01-22 17:11:37 +08:00
Jordan Hrycaj b623909c44
Ledger activate unified accounts cache wrapper (#1939)
* Activate `LedgerRef` wrapper for `AccountsCache`

details:
  `accounts_cache.nim` methods are indirectly processed by the wrapper
  methods from `ledger.nim`.

  This works for all sources except `test_state_db.nim` where the source
  `accounts_cache.nim` is included (rather than imported) in order to
  access objects privy to the very source.

* Provide facility to switch to a preselected `LedgerRef` type

details:
  Can be set as suggestion when initialising `CommonRef`

* Update `CoreDb` test suite for better time tracking

details:
+ Allow time logging by pre-defined block intervals
+ Print `CoreDb`/`Ledger`profiling results (if enabled)
2023-12-12 19:12:56 +00:00
jangko 24ba819cd1
More meaningful error message of processBlockBeaconRoot 2023-10-31 12:54:57 +07:00
jangko 0ee448c1eb
Fix ForkID calculation 2023-10-25 11:10:01 +07:00
jangko c005281391
processBeaconBlockRoot in TxPool(EIP-4788) 2023-10-19 07:50:07 +07:00
andri lim b8e3fc5859
Call processBeaconBlockRoot before processing txs(EIP-4788) (#1822) 2023-10-19 06:55:50 +07:00
andri lim 34b1e29ac9
Fix test_blockchain_json and pyspec simulator for Cancun (#1805)
* Fix test_blockchain_json and pyspec simulator for Cancun

* Preserve applyDeletes comments

* Fix redefinition error

* Move test_macro to tests folder
2023-10-05 10:04:12 +07:00
andri lim 12f494e669
Only one place to declare gwei (#1802)
* Only one place to declare gwei

* Fix weiAmount overflow
2023-10-04 10:47:18 +07:00
andri lim dc1dcfb206
EIP-4844: Fix excessBlobGas handling (#1771)
* EIP-4844: Fix excessBlobGas handling

* Change vmState.difficulty to vmState.difficultyOrPrevRandao
2023-09-24 22:25:41 +07:00
jangko cc7a7db74b
EIP-4788: Make it clear what is an EVM system call 2023-09-22 17:21:09 +07:00
Jordan Hrycaj 6bc55d4e6f
Core db aristo and kvt updates preparing for integration (#1760)
* Kvt: Implemented multi-descriptor access on the same backend

why:
  This behaviour mirrors the one of Aristo and can be used for
  simultaneous transactions on Aristo + Kvt

* Kvt: Update database iterators

why:
  Forgot to run on the top layer first

* Kvt: Misc fixes

* Aristo, use `openArray[byte]` rather than `Blob` in prototype

* Aristo, by default hashify right after cloning descriptor

why:
  Typically, a completed descriptor is expected after cloning. Hashing
  can be suppressed by argument flag.

* Aristo provides `replicate()` iterator, similar to legacy `replicate()`

* Aristo API fixes and updates

* CoreDB: Rename `legacy_persistent` => `legacy_rocksdb`

why:
  More systematic, will be in line with Aristo DB which might have
  more than one persistent backends

* CoreDB: Prettify API sources

why:
  Better to read and maintain

details:
  Annotating with custom pragmas which cleans up the prototypes

* CoreDB: Update MPT/put() prototype allowing `CatchableError`

why:
  Will be needed for Aristo API (legacy is OK with `RlpError`)
2023-09-18 21:20:28 +01:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
andri lim 96fb355efe
Implement EIP-4788: Beacon block root in the EVM (#1722)
* Implement EIP-4788: Beacon block root in the EVM

* EIP-4788: Fix genesis.parentBeaconBlockRoot initialization
2023-08-30 23:29:48 +07:00
andri lim 91704cd3ae
Add current computation to each EVM tracer interface (#1712) 2023-08-25 16:07:20 +07:00
jangko 7a1725e744
Remove EVMFork comparison from eip1559TxNormalization 2023-08-22 13:24:39 +07:00
jangko 92713ef326
Fix rpc.sendRawTransaction and txPool: reject invalid transaction earlier 2023-08-21 09:11:10 +07:00
Jordan Hrycaj 4c9141ffac
Aristo db implement filter serialisation for storage (#1695)
* Remove concept of empty/blind filters

why:
  Not needed. A non-existent filter is is coded as a nil reference.

* Slightly generalised backend iterators

why:
 * VertexID as key for the ID generator state makes no sense
 * there will be more tables addressed by non-VertexID keys

* Store serialised/blobified vertices on memory backend

why:
  This is more in line with the RocksDB backend so more appropriate
  for testing when comparing behaviour. For a speedy memory database,
  a backend-less variant should be used.

* Drop the `Aristo` prefix from names `AristoLayerRef`, etc.

* Suppress compiler warning

why:
  duplicate imports

* Add filter serialisation transcoder

why:
  Will be used as storage format
2023-08-18 20:46:55 +01:00
jangko 0c1236756d
EIP-4895: add withdrawal processing in txpool 2023-08-16 21:28:03 +07:00
andri lim bdaeedb09f
rename data gas to blob gas (#1659)
* rename data gas to blob gas

* bump more submodules
* extend evmc tx_context with EIP-4844 blob_hashes
2023-08-04 19:43:30 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
jangko 2446c22138
pyspec sim: add test cases 2023-07-19 08:28:49 +07:00
jangko ff1a45e095
fix shanghai withdrawal validation
previously, the withdrawal validation is in process_block only,
but the one in persist block, which is also used in synchronizer
is not validated properly.
2023-06-26 07:46:09 +07:00
andri lim 26a8759c34
implementation of EIP-4844: Shard Blob Transactions (#1440)
* EIP-4844: add pointEvaluation precompiled contract

* EIP-4844: validate transaction and block header

* EIP-4844: implement DataHash Op Code

* EIP-4844: txPool support excessDataGas calculation

* EIP-4844: make sure tx produce correct txHash

* EIP-4844: node should not automatically broadcast blob tx to it's peers

* EIP-4844: add test cases

* EIP-4844: add EIP-4844 support to t8n tool

* EIP-4844: update nim-eth to branch eip-4844

* fix t8n transaction decoding

* add t8n test data

* EIP-4844: fix blobHash opcode

* disable blobHash test when evmc_enable
2023-06-24 20:56:44 +07:00
jangko 6544adf360
implement EIP-1153: Transient storage
new EVM opcodes:
- TLOAD  0xb3
- TSTORE 0xb4
2023-06-23 14:04:36 +07:00
jangko 10aabd8ca1
bubble up validate transaction error message 2023-06-12 12:58:53 +07:00