Commit Graph

76 Commits

Author SHA1 Message Date
Jordan Hrycaj 14c3772545
On demand mpt revisited (#2426)
* rebased from `github/on-demand-mpt`

ackn:
  wip: on-demand mpt construction

  Given that actual data is stored in the `Vertex` structure, it's useful
  to think of the MPT as a cache for computing roots rather than being a
  functional requirement on its own.

  This PR engenders this line of thinking by incrementally computing the
  MPT only when it's needed, ie when a state (or similar) root is needed.

  This has the effect of siginficantly reducing memory usage as well as
  improving performance:

  * no need for dirty-mpt-node book-keeping
  * no need to build complex forest of upcoming hashing work
  * only hashes that are functionally needed are ever computed -
  intermediate nodes whose MTP root is not observed are never computed /
  processed

* Unit test hot fixes

* Unit test hot fixes cont.

(somehow lost that part)

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-28 15:03:12 +00:00
Jordan Hrycaj 61bbf40014
Update storage tree admin (#2419)
* Tighten `CoreDb` API for accounts

why:
  Apart from cruft, the way to fetch the accounts state root via a
  `CoreDbColRef` record was unnecessarily complicated.

* Extend `CoreDb` API for accounts to cover storage tries

why:
  In future, this will make the notion of column objects obsolete. Storage
  trees will then be indexed by the account address rather than the vertex
  ID equivalent like a `CoreDbColRef`.

* Apply new/extended accounts API to ledger and tests

details:
  This makes the `distinct_ledger` module obsolete

* Remove column object constructors

why:
  They were needed as an abstraction of MPT sub-trees including storage
  trees. Now, storage trees are handled by the account (e.g. via address)
  they belong to and all other trees can be identified by a constant well
  known vertex ID. So there is no need for column objects anymore.

  Still there are some left-over column object methods wnich will be
  removed next.

* Remove `serialise()` and `PayloadRef` from default Aristo API

why:
  Not needed. `PayloadRef` was used for unstructured/unknown payload
  formats (account or blob) and `serialise()` was used for decodng
  `PayloadRef`. Now it is known in advance what the payload looks
  like.

* Added query function `hasStorageData()` whether a storage area exists

why:
  Useful for supporting `slotStateEmpty()` of the `CoreDb` API

* In the `Ledger` replace `storage.stateEmpty()` by 	`slotStateEmpty()`

* On Aristo, hide the storage root/vertex ID in the `PayloadRef`

why:
  The storage vertex ID is fully controlled by Aristo while the
  `AristoAccount` object is controlled by the application. With the
  storage root part of the `AristoAccount` object, there was a useless
  administrative burden to keep that storage root field up to date.

* Remove cruft, update comments etc.

* Update changed MPT access paradigms

why:
  Fixes verified proxy tests

* Fluffy cosmetics
2024-06-27 09:01:26 +00:00
andri lim 27339e9520
Simplify txpool baseFeeGet (#2416)
* Simplify txpool baseFeeGet

- Avoid using toEVMFork because we are not in EVM
- Rename `isLondon` to `isLondonOrLater`

* Remove timestamp from isLondonOrLater
2024-06-27 12:54:36 +07:00
andri lim 99ff8dc876
Fix t8n: blobGasUsed exceeds allowance issue (#2407)
* Fix t8n: blobGasUsed exceeds allowance issue

* Put blobGasUsed validation into transaction precessing pipeline
2024-06-24 07:56:24 +02:00
Jacek Sieka 6b68ff92d3
Allocation-free nibbles buffer (#2406)
This buffer eleminates a large part of allocations during MPT traversal,
reducing overall memory usage and GC pressure.

Ideally, we would use it throughout in the API instead of
`openArray[byte]` since the built-in length limit appropriately exposes
the natural 64-nibble depth constraint that `openArray` fails to
capture.
2024-06-22 22:33:37 +02:00
Jordan Hrycaj 081cb15493
Coredb maintenance (#2398)
* CoreDb: remove PHK tries

why:
  There is no general use anymore for an MPT with a pre-hashed key. It
  was used to resemble the `SecureHexaryTrie` logic from the legacy DB.

  The only pace where this is needed is the `Leger` which uses a
  a distinct MPT version anyway (see `distinct_ledgers.nim`.)

* Rename `CoreDx*` -> `CoreDb*`

why:
  The naming `CoreDx*` was used to differentiate the new CoreDb API from
  the legacy API which had descriptors named `CoreDb*`.
2024-06-19 14:13:12 +00:00
andri lim 61a809cf4d
Remove EVM indirect imports and unused EVM errors (#2370)
Those indirect imports are used when there was two EVMs.
2024-06-17 09:56:39 +02:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Barnabas Busa 23027baf30
fix: migration of ethereum-package to ethpandaops (#2348)
* fix: migration of ethereum-package to ethpandaops

* add hidden files too
2024-06-13 16:03:19 +02:00
web3-developer db8c5b90bd
Cleanup stateless and block witness code. (#2295)
* Cleanup unneeded stateless and block witness code. Keeping MultiKeys which is used in the eth_getProofsByBlockNumber RPC endpoint which is needed for the Fluffy state network bridge.

* Rename generateWitness flag to collectWitnessData to better describe what the flag does. We only collect the keys of the touched accounts and storage slots but no block witness generation is supported for now.

* Move remaining stateless code into nimbus directory.

* Add vmstate parameter to ChainRef to fix test.

* Exclude *.in from check copyright year

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-06-08 15:05:00 +07:00
tersec fd03038cab
Replace some usage of std/options with results Opt (#2323)
* Replace some usage of std/options with results Opt

* more updates
2024-06-07 23:39:58 +02:00
andri lim b3a5c67532
Remove exceptions from EVM (#2314)
* Remove exception from evm memory

* Remove exception from gas meter

* Remove exception from stack

* Remove exception from precompiles

* Remove exception from gas_costs

* Remove exception from op handlers

* Remove exception from op dispatcher

* Remove exception from call_evm

* Remove exception from EVM

* Fix tools and tests

* Remove exception from EVMC

* fix evmc

* Fix evmc

* Remove remnants of async evm stuff

* Remove superflous error handling

* Proc to func

* Fix errors detected by CI

* Fix EVM op call stack usage

* REmove exception handling from getVmState

* Better error message instead of just doAssert

* Remove unused validation

* Remove superflous catchRaise

* Use results.expect instead of unsafeValue
2024-06-07 15:24:32 +07:00
Jordan Hrycaj e9eae4df70
Core db disable legacy api n remove distinct tries (#2299)
* CoreDb: Remove crufty second/off-site KVT

why:
  Was used to allow late `Clique` to store directly to disk

* CoreDb: Remove prune flag related functionality

why:
  Is completely legacy stuff

* CoreDb: Remove dependence on legacy API (tests unsupported yet)

why:
  Does not fully support Aristo

* Re-factoring `state_db` using new API

details:
  Only minimum changes needed to compile `nimbus`

* Update tests and aux modules

* Turn off legacy API and remove `distinct_tries`

comment:
  The legacy API has now cruft status, will be removed soon

* Fix copyright years

* Update rpc for verified proxy

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-05 20:52:04 +00:00
Jacek Sieka 7f76586214
Speed up account ledger a little (#2279)
`persist` is a hotspot when processing blocks because it is run at least
once per transaction and loops over the entire account cache every time.

Here, we introduce an extra `dirty` map that keeps track of all accounts
that need checking during `persist` which fixes the immediate
inefficiency, though probably this could benefit from a more thorough
review - we also get rid of the unused clearCache flag - we start with
a fresh cache on every fresh vmState.

* avoid unnecessary code hash comparisons
* avoid unnecessary copies when iterating
* use EMPTY_CODE_HASH throughout for code hash comparison
2024-06-02 21:21:29 +02:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
andri lim eaf3d9897e
Simplify AccountsLedgerRef complexity (#2239) 2024-05-29 13:06:49 +02:00
tersec 709200f62a
remove expicit PoW support from tx pool; tighten tx pool exceptions specs (#2235) 2024-05-28 20:26:51 +02:00
Etan Kissling 741fcca9b6
Workaround for disallowed transaction superseding (#2181)
* Workaround for disallowed transaction superseding

The transaction spammer from Kurtosis keeps spamming transactions with
the same nonce because report 'pending' account nocne based on 'latest'
rather than actually considering the mempool. To avoid errors from
rejecting those, disable the required gas price bump when replacing
a transaction with a new nonce.

* Lint

* Skip superseding negative test
2024-05-28 13:38:57 +07:00
tersec e895c0baeb
rm Clique consensus method support and Goerli network (#2219)
* rm Clique consensus method support and Goerli network

* rm a few more SealingEngineRef and GoerliNets
2024-05-25 16:12:14 +02:00
Jordan Hrycaj ee9aea171d
Culling legacy DB and accounts cache (#2197)
details:
+ Compiles nimbus all_tests
+ Failing tests have been commented out
2024-05-20 10:17:51 +00:00
andri lim 38eaebc5c7
TxPool: rank calculator now include profit from blob fee (#2196)
* TxPool: rank calculator now include profit from blob fee

* Fix copyright year
2024-05-20 14:30:48 +07:00
Etan Kissling c4c37302b1
Introduce wrapper type for EIP-4844 transactions (#2177)
* Introduce wrapper type for EIP-4844 transactions

EIP-4844 blob sidecars are a concept that only exists in the mempool.
After inclusion of a transaction into an execution block, only the
versioned hash within the transaction remains. To improve type safety,
replace the `Transaction.networkPayload` member with a wrapper type
`PooledTransaction` that is used in contexts where blob sidecars exist.

* Bump nimbus-eth2 to 87605d08a7f9cfc3b223bd32143e93a6cdf351ac

* IPv6 'listen-address' in `nimbus_verified_proxy`

* Bump nim-libp2p to 21cbe3a91a70811522554e89e6a791172cebfef2

* Fix beacon_lc_bridge payload conversion and conf.listenAddress type

* Change nimbus_verified_proxy.asExecutionData param to SomeExecutionPayload

* Rerun nph to fix asExecutionData style format

* nimbus_verified_proxy listenAddress

* Use PooledTransaction in nimbus-eth1 tests

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-05-15 10:07:59 +07:00
andri lim 47605dbdf3
Fix txPool: excessBlobGas should be calculated before entering EVM, not after (#2149) 2024-04-24 07:50:50 +07:00
andri lim 52aa87c539
Align with EIP-4844: rename getBlobGasPrice to getBlobBaseFee (#2043) 2024-02-21 16:14:34 +07:00
andri lim 7c1af9a78f
Add style check to config.nims and fix styles in source code (#2038)
* Add style check to config.nims and fix styles in source code

* Fix copyright year
2024-02-20 10:07:38 +07:00
andri lim bea558740f
Reduce compiler warnings (#2030)
* Reduce compiler warnings

* Reduce compiler warnings in test code
2024-02-16 16:08:07 +07:00
andri lim 966adcb124
Prepare source code for nim v2 CI (#2028)
* Prepare source code for nim v2 CI

* Fix copyright year
2024-02-15 09:57:05 +07:00
Jordan Hrycaj b623909c44
Ledger activate unified accounts cache wrapper (#1939)
* Activate `LedgerRef` wrapper for `AccountsCache`

details:
  `accounts_cache.nim` methods are indirectly processed by the wrapper
  methods from `ledger.nim`.

  This works for all sources except `test_state_db.nim` where the source
  `accounts_cache.nim` is included (rather than imported) in order to
  access objects privy to the very source.

* Provide facility to switch to a preselected `LedgerRef` type

details:
  Can be set as suggestion when initialising `CommonRef`

* Update `CoreDb` test suite for better time tracking

details:
+ Allow time logging by pre-defined block intervals
+ Print `CoreDb`/`Ledger`profiling results (if enabled)
2023-12-12 19:12:56 +00:00
jangko b0000eed8b
Add check copyright year linter to CI 2023-11-01 10:41:20 +07:00
jangko 4febd1899f
TxPool: Bubble up error from packer to assembleBlock 2023-11-01 09:24:32 +07:00
jangko b74a080a3d
TxPool will not pack tx with invalid chainId 2023-11-01 08:56:57 +07:00
jangko 0ee448c1eb
Fix ForkID calculation 2023-10-25 11:10:01 +07:00
andri lim 77289c7795
More cancun tests (#1843)
* Engine API simulator: More Cancun tests

* Fix Cancun validation in Engine API and TxPool
2023-10-23 20:59:57 +07:00
andri lim 91106ec6bf
Fix txPool: Limit number of blobs packed into a block(EIP-4844) (#1839) 2023-10-22 15:28:28 +07:00
andri lim 7169c846a3
Fix TxPool when handling EIP-4844 blob tx (#1831)
* Fix TxPool when handling EIP-4844 blob tx
2023-10-20 15:30:05 +07:00
jangko c005281391
processBeaconBlockRoot in TxPool(EIP-4788) 2023-10-19 07:50:07 +07:00
andri lim 04c7ed8ec9
Change EthTime from std.Time to distinct uint64 (#1820)
* Change EthTime from std.Time to distinct uint64

* Bump nimbus-eth2
2023-10-18 09:16:11 +07:00
andri lim 12f494e669
Only one place to declare gwei (#1802)
* Only one place to declare gwei

* Fix weiAmount overflow
2023-10-04 10:47:18 +07:00
jangko 369a54d62a
Add makeNextWithdrawals to engine api simulator 2023-09-30 20:16:45 +07:00
andri lim 501d8a369a
Add BeaconSync reorg tests (#1782)
* Add BeaconSync reorg tests

* Fix redefinition error in tx_sender.nim
2023-09-30 19:20:29 +07:00
andri lim dc1dcfb206
EIP-4844: Fix excessBlobGas handling (#1771)
* EIP-4844: Fix excessBlobGas handling

* Change vmState.difficulty to vmState.difficultyOrPrevRandao
2023-09-24 22:25:41 +07:00
Jordan Hrycaj 6bc55d4e6f
Core db aristo and kvt updates preparing for integration (#1760)
* Kvt: Implemented multi-descriptor access on the same backend

why:
  This behaviour mirrors the one of Aristo and can be used for
  simultaneous transactions on Aristo + Kvt

* Kvt: Update database iterators

why:
  Forgot to run on the top layer first

* Kvt: Misc fixes

* Aristo, use `openArray[byte]` rather than `Blob` in prototype

* Aristo, by default hashify right after cloning descriptor

why:
  Typically, a completed descriptor is expected after cloning. Hashing
  can be suppressed by argument flag.

* Aristo provides `replicate()` iterator, similar to legacy `replicate()`

* Aristo API fixes and updates

* CoreDB: Rename `legacy_persistent` => `legacy_rocksdb`

why:
  More systematic, will be in line with Aristo DB which might have
  more than one persistent backends

* CoreDB: Prettify API sources

why:
  Better to read and maintain

details:
  Annotating with custom pragmas which cleans up the prototypes

* CoreDB: Update MPT/put() prototype allowing `CatchableError`

why:
  Will be needed for Aristo API (legacy is OK with `RlpError`)
2023-09-18 21:20:28 +01:00
andri lim 96fb355efe
Implement EIP-4788: Beacon block root in the EVM (#1722)
* Implement EIP-4788: Beacon block root in the EVM

* EIP-4788: Fix genesis.parentBeaconBlockRoot initialization
2023-08-30 23:29:48 +07:00
jangko 7a1725e744
Remove EVMFork comparison from eip1559TxNormalization 2023-08-22 13:24:39 +07:00
jangko 92713ef326
Fix rpc.sendRawTransaction and txPool: reject invalid transaction earlier 2023-08-21 09:11:10 +07:00
jangko 4fd53156f5
Fix engine api: getPayload V2 and V3 now returns correct blockValue 2023-08-18 14:25:11 +07:00
jangko 0c1236756d
EIP-4895: add withdrawal processing in txpool 2023-08-16 21:28:03 +07:00
andri lim bdaeedb09f
rename data gas to blob gas (#1659)
* rename data gas to blob gas

* bump more submodules
* extend evmc tx_context with EIP-4844 blob_hashes
2023-08-04 19:43:30 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
jangko f02c20c1f9
fix EIP-4844: txpool missing dataGasUsed when generate block header 2023-07-25 12:28:17 +07:00