Commit Graph

345 Commits

Author SHA1 Message Date
Jordan Hrycaj 800fd77333
Core db remove legacy phrases (#2468)
* Rename `newKvt()` -> `ctx.getKvt()`

why:
  Clean up legacy shortcut. Also, the `KVT` returned is not instantiated
  but refers to the shared `KVT` that resides in a context which is a
  generalisation of an in-memory database fork. The function `ctx`
  retrieves the default context.

* Rename `newTransaction()` -> `ctx.newTransaction()`

why:
  Clean up legacy shortcut. The transaction is applied to a context as a
  generalisation of an in-memory database fork. The function `ctx`
  retrieves the default context.

* Rename `getColumn(CtGeneric)` -> `getGeneric()`

why:
  No more a list of well known sub-tries needed, a single one is enough.
  In fact, `getColumn()` did only support a single sub-tree by now.

* Reduce TODO list
2024-07-10 12:19:35 +00:00
andri lim f04f30c72b
Reduce EVM complexity by removing forkOverride (#2448)
* Reduce EVM complexity by removing forkOverride

* Fixes
2024-07-04 15:48:36 +02:00
andri lim c24affadee
Use simpler schema when writing transactions, receipts, and withdrawals (#2420)
* Use simpler schema when writing transactions, receipts, and withdrawals

Using MPT not only slow but also take up more spaces than needed.
Aristo will remove older tries and only keep the last block tries.
Using simpler schema will avoid those problems.

* Rename getTransaction to getTransactionByIndex
2024-06-29 12:43:17 +07:00
andri lim 27339e9520
Simplify txpool baseFeeGet (#2416)
* Simplify txpool baseFeeGet

- Avoid using toEVMFork because we are not in EVM
- Rename `isLondon` to `isLondonOrLater`

* Remove timestamp from isLondonOrLater
2024-06-27 12:54:36 +07:00
Jacek Sieka 768307d91d
Cache code and invalid jump destination tables (fixes #2268) (#2404)
It is common for many accounts to share the same code - at the database
level, code is stored by hash meaning only one copy exists per unique
program but when loaded in memory, a copy is made for each account.

Further, every time we execute the code, it must be scanned for invalid
jump destinations which slows down EVM exeuction.

Finally, the extcodesize call causes code to be loaded even if only the
size is needed.

This PR improves on all these points by introducing a shared
CodeBytesRef type whose code section is immutable and that can be shared
between accounts. Further, a dedicated `len` API call is added so that
the EXTCODESIZE opcode can operate without polluting the GC and code
cache, for cases where only the size is requested - rocksdb will in this
case cache the code itself in the row cache meaning that lookup of the
code itself remains fast when length is asked for first.

With 16k code entries, there's a 90% hit rate which goes up to 99%
during the 2.3M attack - the cache significantly lowers memory
consumption and execution time not only during this event but across the
board.
2024-06-21 09:44:10 +02:00
andri lim 61a809cf4d
Remove EVM indirect imports and unused EVM errors (#2370)
Those indirect imports are used when there was two EVMs.
2024-06-17 09:56:39 +02:00
andri lim 69044dda60
Remove AccountStateDB (#2368)
* Remove AccountStateDB

AccountStateDB should no longer be used.
It's usage have been reduce to read only operations.
Replace it with LedgerRef to reduce maintenance burden.

* remove extra spaces

Co-authored-by: tersec <tersec@users.noreply.github.com>

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-16 10:21:02 +07:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Jacek Sieka 189a20bbae
Avoid recomputing hashes when persisting data (#2350) 2024-06-14 07:10:00 +02:00
Barnabas Busa 23027baf30
fix: migration of ethereum-package to ethpandaops (#2348)
* fix: migration of ethereum-package to ethpandaops

* add hidden files too
2024-06-13 16:03:19 +02:00
Jacek Sieka c48b527eea
simplify error handling in block processing (#2337)
* ValidationResult -> Result
* get rid of mixed exception / other styles
2024-06-11 17:50:22 +02:00
Jacek Sieka 359f7ada65
eth: avoid sink (#2331)
* eth: avoid sink

* bump

* fix extra transactions

from #2330
2024-06-10 14:16:22 +07:00
Jacek Sieka 0b32078c4b
Consolidate block type for block processing (#2325)
This PR consolidates the split header-body sequences into a single EthBlock
sequence and cleans up the fallout from that which significantly reduces
block processing overhead during import thanks to less garbage collection
and fewer copies of things all around.

Notably, since the number of headers must always match the number of bodies,
we also get rid of a pointless degree of freedom that in the future could
introduce unnecessary bugs.

* only read header and body from era file
* avoid several unnecessary copies along the block processing way
* simplify signatures, cleaning up unused arguemnts and returns
* use `stew/assign2` in a few strategic places where the generated
  nim assignent is slow and add a few `move` to work around poor
  analysis in nim 1.6 (will need to be revisited for 2.0)

```
stats-20240607_2223-a814aa0b.csv vs stats-20240608_0714-21c1d0a9.csv
                       bps_x     bps_y     tps_x        tps_y    bpsd    tpsd    timed
block_number
(498305, 713245]    1,540.52  1,809.73  2,361.58  2775.340189  17.63%  17.63%  -14.92%
(713245, 928185]      730.36    865.26  1,715.90  2028.973852  18.01%  18.01%  -15.21%
(928185, 1143126]     663.03    789.10  2,529.26  3032.490771  19.79%  19.79%  -16.28%
(1143126, 1358066]    393.46    508.05  2,152.50  2777.578119  29.13%  29.13%  -22.50%
(1358066, 1573007]    370.88    440.72  2,351.31  2791.896052  18.81%  18.81%  -15.80%
(1573007, 1787947]    283.65    335.11  2,068.93  2441.373402  17.60%  17.60%  -14.91%
(1787947, 2002888]    287.29    342.11  2,078.39  2474.179448  18.99%  18.99%  -15.91%
(2002888, 2217828]    293.38    343.16  2,208.83   2584.77457  17.16%  17.16%  -14.61%
(2217828, 2432769]    140.09    167.86  1,081.87  1296.336926  18.82%  18.82%  -15.80%

blocks: 1934464, baseline: 3h13m1s, contender: 2h43m47s
bpsd (mean): 19.55%
tpsd (mean): 19.55%
Time (total): -29m13s, -15.14%
```
2024-06-09 16:32:20 +02:00
web3-developer db8c5b90bd
Cleanup stateless and block witness code. (#2295)
* Cleanup unneeded stateless and block witness code. Keeping MultiKeys which is used in the eth_getProofsByBlockNumber RPC endpoint which is needed for the Fluffy state network bridge.

* Rename generateWitness flag to collectWitnessData to better describe what the flag does. We only collect the keys of the touched accounts and storage slots but no block witness generation is supported for now.

* Move remaining stateless code into nimbus directory.

* Add vmstate parameter to ChainRef to fix test.

* Exclude *.in from check copyright year

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-06-08 15:05:00 +07:00
andri lim b3a5c67532
Remove exceptions from EVM (#2314)
* Remove exception from evm memory

* Remove exception from gas meter

* Remove exception from stack

* Remove exception from precompiles

* Remove exception from gas_costs

* Remove exception from op handlers

* Remove exception from op dispatcher

* Remove exception from call_evm

* Remove exception from EVM

* Fix tools and tests

* Remove exception from EVMC

* fix evmc

* Fix evmc

* Remove remnants of async evm stuff

* Remove superflous error handling

* Proc to func

* Fix errors detected by CI

* Fix EVM op call stack usage

* REmove exception handling from getVmState

* Better error message instead of just doAssert

* Remove unused validation

* Remove superflous catchRaise

* Use results.expect instead of unsafeValue
2024-06-07 15:24:32 +07:00
Jordan Hrycaj e9eae4df70
Core db disable legacy api n remove distinct tries (#2299)
* CoreDb: Remove crufty second/off-site KVT

why:
  Was used to allow late `Clique` to store directly to disk

* CoreDb: Remove prune flag related functionality

why:
  Is completely legacy stuff

* CoreDb: Remove dependence on legacy API (tests unsupported yet)

why:
  Does not fully support Aristo

* Re-factoring `state_db` using new API

details:
  Only minimum changes needed to compile `nimbus`

* Update tests and aux modules

* Turn off legacy API and remove `distinct_tries`

comment:
  The legacy API has now cruft status, will be removed soon

* Fix copyright years

* Update rpc for verified proxy

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-05 20:52:04 +00:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
andri lim 74cc3b6127
Implement engine_getClientVersionV1 (#2233)
* Implement engine_getClientVersionV1

* full git revision string

* Limit GitRevisionString to 8 chars

* Fixes

* Debug windows CI

* debug windows ci

* produce git revision using -C

* try not to delete .git folder in windows ci

* Harden GitRevision procuration

* Add double quotes to git -C param

* Escape sourcePath

* Remove double quotes from git -C param
2024-05-29 09:20:50 +02:00
Etan Kissling b7a7745e24
Workaround for 0 gas price estimation (#2180)
* Workaround for 0 gas price estimation

The transaction spammer from Kurtosis estimates the gas price of its
transactions using 'eth_gasPrice'. Our implementation returns 0 when
no transactions have been executed yet, not taking into account the
EIP-1559 block base fee. Force a hardcoded minimum for now to unstuck.

* Adjust tests to cover new minimum gas fee

* Skip gas price test
2024-05-28 13:36:38 +07:00
Jordan Hrycaj ee9aea171d
Culling legacy DB and accounts cache (#2197)
details:
+ Compiles nimbus all_tests
+ Failing tests have been commented out
2024-05-20 10:17:51 +00:00
Etan Kissling c4c37302b1
Introduce wrapper type for EIP-4844 transactions (#2177)
* Introduce wrapper type for EIP-4844 transactions

EIP-4844 blob sidecars are a concept that only exists in the mempool.
After inclusion of a transaction into an execution block, only the
versioned hash within the transaction remains. To improve type safety,
replace the `Transaction.networkPayload` member with a wrapper type
`PooledTransaction` that is used in contexts where blob sidecars exist.

* Bump nimbus-eth2 to 87605d08a7f9cfc3b223bd32143e93a6cdf351ac

* IPv6 'listen-address' in `nimbus_verified_proxy`

* Bump nim-libp2p to 21cbe3a91a70811522554e89e6a791172cebfef2

* Fix beacon_lc_bridge payload conversion and conf.listenAddress type

* Change nimbus_verified_proxy.asExecutionData param to SomeExecutionPayload

* Rerun nph to fix asExecutionData style format

* nimbus_verified_proxy listenAddress

* Use PooledTransaction in nimbus-eth1 tests

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-05-15 10:07:59 +07:00
Etan Kissling 1b8f2b0ea5
Workaround for missing implementation of 'pending' tag (#2178)
The transaction spammer from Kurtosis requests the nonce value of its
account based on 'pending' tag. As we lack that implementation, go with
the next best answer from the 'latest' tag. This does not include info
about transactions from the mempool that have not yet been executed.
2024-05-13 16:42:28 +03:00
andri lim 6694e240d7
Repositioning blob hash validation in newPayload of engine API (#2141) 2024-04-20 02:43:13 +07:00
jangko 160e033663
Remove intermediate RpcCallData 2024-04-16 19:52:07 +07:00
andri lim e713f3c287
Implement eth_feeHistory (#2130)
* Implement eth_feeHistory

* Fix copyright year
2024-04-16 08:02:42 +07:00
andri lim 516dd95566
Implement eth_blobBaseFee (#2124) 2024-04-04 07:25:56 +07:00
andri lim 0d4db21dad
Add engine_getPayloadV4 and engine_newPayloadV4 (#2116) 2024-03-28 18:59:23 +07:00
andri lim eb67e78fde
Implement RPC methods: debug_getRawBlock, debug_getRawHeader, debug_g… (#2098)
* Implement RPC methods: debug_getRawBlock, debug_getRawHeader, debug_getRawReceipts, debug_getRawTransaction

* Fix engine api sim when calling setupDebugRpc
2024-03-22 23:40:16 +07:00
andri lim 7ea6d719d9
Implement RPC method eth_getAccessList (#2091)
* Implement RPC method eth_getAccessList

* Fix comment
2024-03-21 18:24:32 +07:00
andri lim 30277be1f3
Bump nim-web3 to 285d97c2b05bbe2a13dab4b52ea878157fb1a1a1 (#2088)
* Bump nim-web3 to 285d97c2b05bbe2a13dab4b52ea878157fb1a1a1

Unify EthCall/EthSend into TransactionArgs (#138)

* bump ssz-serialization

* Fix BlockNumber conversion

* Bump ssz-serialization: Restrict toSszType usage to non SszType in readSszBytes (#81)
2024-03-21 08:05:22 +07:00
andri lim 6ff2edc416
Fix styles (#2046)
* Fix styles

* Fix copyright year
2024-02-21 23:04:59 +07:00
andri lim 52aa87c539
Align with EIP-4844: rename getBlobGasPrice to getBlobBaseFee (#2043) 2024-02-21 16:14:34 +07:00
andri lim 7b4ef814ea
Bump eth_tests and related fixes (#2042)
* Bump eth_tests and related fixes

* Fix tests
2024-02-21 16:14:20 +07:00
andri lim 7c1af9a78f
Add style check to config.nims and fix styles in source code (#2038)
* Add style check to config.nims and fix styles in source code

* Fix copyright year
2024-02-20 10:07:38 +07:00
andri lim bea558740f
Reduce compiler warnings (#2030)
* Reduce compiler warnings

* Reduce compiler warnings in test code
2024-02-16 16:08:07 +07:00
andri lim 966adcb124
Prepare source code for nim v2 CI (#2028)
* Prepare source code for nim v2 CI

* Fix copyright year
2024-02-15 09:57:05 +07:00
web3-developer 332d294e77
Storage of block witnesses (#1986)
* Added procs to get and store block witness in db and add generate-witness cli flag.

* Completed initial implementation of block witness storage.

* Added test to verify witness is persisted to db after call to persistBlock.

* Update getBlockWitness to return witness using Result type.

* Make generate witness parameter hidden.
2024-02-13 17:49:41 +08:00
web3-developer 93fb4c8f59
More witness fixes (#2009)
* Update experimental rpc test to do further validation on proof responses.

* Enable zero value storage slots in the witness cache data so that proofs will be returned when a storage slot is updated to zero. Refactor and simplify implementation of getProofs endpoint.

* Improve test validation.

* Minor fixes and added test to track the changes introduced in every block using a local state.

* Refactor and cleanup test.

* Comments added to test and account cache fixes applied to account ledger.

* Return updated storage slots even when storage is empty and add test to build.

* Fix copyright and remove incorrect depth check during witness building in writeShortRlp assertion.
2024-02-09 12:09:02 +08:00
web3-developer a5ac5d3078
Premix persist tool improvements (#2000)
* Add premix persist tool to Makefile to build as separate binary.

* Fix rootBytes.len > 0 assertion defect in HexaryTrie by adding call to com.db.compensateLegacySetup().

* Run hardForkTransition before executing transactions in getBlockWitness.

* Improve logging and add help message.

* Retry requests for block data.
2024-02-01 22:11:41 +08:00
andri lim c635e160d9
Implement combo http server for rpc, engine_api, and graphql services (#1992)
* Combo HTTP server implementation

* Use json flavor for jwt_auth decoder
2024-01-29 20:20:04 +07:00
web3-developer 54644fa3cb
Witness Generation Bug Fixes (#1981)
* Return slots when verifying witness regardless of code length.

* Prevent AccountCache WitnessData codeTouched from being reset.

* Default to using {wfNoFlag} for witness flags in tests to allow running with data from before EIP170.

* Add additional json files to experimental JSON RPC test.

* Use HTTP RPC server in tests.

* Add test to check that block witness contains bytecode.
2024-01-25 01:18:45 +08:00
andri lim 31c288d5e5
Implement JSON-RPC method: eth_getBlockReceipts (#1984)
* Implement JSON-RPC method: eth_getBlockReceipts

* Increment index in eth_getBlockReceipts
2024-01-24 18:04:59 +07:00
web3-developer 48630ccb03
Experimental RPC endpoints for generating block witnesses (#1977)
* Completed draft implementation of witness JSON-RPC endpoints for portal network bridge.

* Updated Nimbus RPC configuration to support enabling experimental endpoints.

* Moved witness verification tests.

* Added json test for getProof.

* Added main procs to new tests to fix test suite.

* Added getBlockWitness test to blockchain json test suite.

* Added tests for experimental RPC endpoints and improved the API to support returning state proofs from before or after block execution.

* Correctly rollback transaction in getBlockWitness proc.
2024-01-22 17:11:37 +08:00
jangko 3e21281d12
Bump nim-web3 and others
Bump nim-json-rpc and nimbus-eth2 too.
Reason: both nim-json-rpc and nim-web3 migrate from
stdlib/json to nim-json-serialization
2024-01-14 10:41:23 +07:00
web3-developer 79c6bdc214
Implementation of Nimbus eth_getProof RPC Endpoint. (#1960)
* Initial implementation of eth_getProof endpoint.

* Implemented generation of account and storage proofs.

* Minor fixes and additional tests.

* Refactor getBranch code into a separate file.

* Improve usage of test data.

* Fix copyright year.

* Return zero hash for codeHash and storageHash if account doesn't exist.

* Update copyright notice and moved trie key hashing to inside getBranch proc.
2024-01-09 10:05:52 +08:00
jangko c0d52ba179
Get rid of HexQuantityStr and HexDataStr usage
Both types are not safe and require validation/conversion
from rpc implementer.
This PR change it to safer types and delegate the conversion
and validation to the rpc library.
2023-12-27 07:48:53 +07:00
jangko 5fd54961e3
Bump nim-web3: remove rpc types duplicates 2023-12-13 07:58:12 +07:00
jangko 25bc8e4b22
Add blobGasPrice and blobGasUsed to ReceiptObject of RPC 2023-11-02 11:38:07 +07:00
jangko b0000eed8b
Add check copyright year linter to CI 2023-11-01 10:41:20 +07:00
andri lim 77289c7795
More cancun tests (#1843)
* Engine API simulator: More Cancun tests

* Fix Cancun validation in Engine API and TxPool
2023-10-23 20:59:57 +07:00