Commit Graph

3893 Commits

Author SHA1 Message Date
Jacek Sieka 01ab209497
cache account payload (#2478)
Instead of caching just the storage id, we can cache the full payload
which further reduces expensive hikes
2024-07-12 15:08:26 +02:00
Jacek Sieka d07540766f
coredb: tracking fixes (#2476) 2024-07-12 13:40:13 +02:00
Kim De Mey 70682cd4a8
Bump NimYAML module and related changes (#2474) 2024-07-12 09:30:50 +02:00
Advaita Saha 25af347dfd
Shift era helpers to a different file (#2475)
* shift helpers to a different file

* fix: few logic fixed for transition from era1 to era
2024-07-12 03:15:14 +00:00
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Jacek Sieka a6764670f0
merge: avoid hike allocations (#2472)
hike allocations (and the garbage collection maintenance that follows)
are responsible for some 10% of cpu time (not wall time!) at this point
- this PR avoids them by stepping through the layers one step at a time,
simplifying the code at the same time.
2024-07-11 13:26:46 +02:00
Kim De Mey 4a20756e6b
Remove unused seed_db and related code (#2471) 2024-07-10 23:02:15 +02:00
Kim De Mey 94340037bf
Set the routing table ip limits back to defaults (#2470) 2024-07-10 17:26:30 +02:00
Kim De Mey 5ae5bd8b69
Bump portal-mainnet for updated fluffy bootstrap ENRs (#2469) 2024-07-10 15:45:11 +02:00
Jordan Hrycaj 800fd77333
Core db remove legacy phrases (#2468)
* Rename `newKvt()` -> `ctx.getKvt()`

why:
  Clean up legacy shortcut. Also, the `KVT` returned is not instantiated
  but refers to the shared `KVT` that resides in a context which is a
  generalisation of an in-memory database fork. The function `ctx`
  retrieves the default context.

* Rename `newTransaction()` -> `ctx.newTransaction()`

why:
  Clean up legacy shortcut. The transaction is applied to a context as a
  generalisation of an in-memory database fork. The function `ctx`
  retrieves the default context.

* Rename `getColumn(CtGeneric)` -> `getGeneric()`

why:
  No more a list of well known sub-tries needed, a single one is enough.
  In fact, `getColumn()` did only support a single sub-tree by now.

* Reduce TODO list
2024-07-10 12:19:35 +00:00
web3-developer 9fc5495d49
Update nim-rocksdb to latest version. (#2467) 2024-07-10 14:18:02 +08:00
Kim De Mey 54e3fd1a94
Move Portal wire and networks setup to new portal_node module (#2464) 2024-07-09 19:22:25 +02:00
Jacek Sieka 25b5f01357
bump stint (#2465)
avoids extreme modmul bottleneck
2024-07-09 18:07:21 +02:00
Jacek Sieka 3382c2427b
increase rdb cache sizes (#2466)
This trivial bump should improve performance a bit without costing too
much memory - as the trie grows, so does the number of levels in it and
creating hikes becomes ever more expensive - hopefully this cache
increase should give a nice little boost even if it's not a lot.
2024-07-09 17:35:27 +02:00
Jacek Sieka ab23148aab
don't rewrite hash->slot map (#2463)
Avoid writing the same slot/hash values to the hash->slot mapping
to avoid spamming the rocksdb WAL and cause unnecessary compaction

In the same vein, avoid writing trivially detectable A-B-A storage
changes which happen with surprising frequency.
2024-07-09 17:25:43 +02:00
Advaita Saha 9a499eb45f
Era support for nimbus import (#2429)
* add the era-dir option

* feat: support for era files in nimbus import

* fix: metric logs

* fix: eraDir check

* fix: redundant code and sepolia support

* fix: remove dependency from csv + formatting

* fix: typo

* fix: RVO

* fix: parseBiggestInt

* fix: opt impl

* fix: network agnostic loading

* fix: shift to int64
2024-07-09 15:28:01 +02:00
andri lim 4fa3756860
Convert GasInt to uint64, bump nim-eth and nimbus-eth2 (#2461)
* Convert GasInt to uint64, bump nim-eth and nimbus-eth2

* Bump nimbus-eth2

* int64.high.GasInt instead of 0x7fffffffffffffff.GasInt
2024-07-07 06:52:11 +00:00
andri lim e8683692fd
EVM gasSstore refund reduction using positive integer (#2460)
This is the hopefully the last part of preparations
before converting GasInt to uint64
2024-07-06 08:39:38 +07:00
andri lim 4eaae5cbfa
EVM gasCall values always stay on positive side (#2459)
* EVM gasCall values always stay on positive side

This is also another part of preparations before
converting GasInt to uint64

* Fix test_evm_support
2024-07-06 08:39:22 +07:00
andri lim c775c906a2
Fix LedgerRef storage iterator and add test (#2458) 2024-07-05 10:15:48 +00:00
andri lim 6fe7411ac0
Saner EVM gasCosts (#2457)
This is also a part of preparations before converting GasInt to uint64
2024-07-05 11:55:13 +07:00
andri lim 23c00ce88c
Separate evmc gasCosts and nim-evm gasCosts (#2454)
This is part of preparations before converting GasInt to uint64
2024-07-05 07:00:03 +07:00
Jacek Sieka 7d78fd97d5
avoid allocations for slot storage (#2455)
Introduce a new `StoData` payload type similar to `AccountData`

* slightly more efficient storage format
* typed api
* fewer seqs
* fix encoding docs - it wasn't rlp after all :)
2024-07-04 23:48:45 +00:00
tersec 1f40b710ee
fix UnusedImport warnings; bump nim-bearssl, nim-stint, and nim-stew (#2456) 2024-07-05 06:46:59 +07:00
Jacek Sieka 893bfa4305
Enable LTO compilation (#2450)
* Enable LTO compilation

Similar to nimbus-eth2, LTO gives a significant boost for any CPU-bound operations such as the EVM.

The options are copied straight from nimbus-eth2 - for example at block height 1.7M there's a computation-heavy section where we can see a 15%-20% improvement in block processing time.

```
                       bps_x     bps_y     tps_x     tps_y time_x time_y     bpsd     tpsd    timed

(1722223, 1733334]    102.52    138.90  1,049.67  1,420.61  2m41s  1m58s   35.78%   35.78%  -26.32%
```

* avoid defer

When evmc recursion is enabled together with LTO, we run out of stack
space.

`defer` creates an exception handling context that takes up hundreds of
bytes of stack space - now that the EVM is no longer using exceptions,
we can safely get rid of it.
2024-07-04 23:10:40 +07:00
Jacek Sieka 79788c01d4
Add debug mode for disabling per-chunk state root validation (#2453)
This significantly speeds up block import at the cost of less protection
against invalid data, potentially resulting in an invalid database
getting stored.

The risk is small given that import is used only for validated data -
evaluating the right level of of validation vs performance is left for a
future PR.

A side effect of this approach is that there is no cached stated root in
the database - computing it currently requires a lot of memory since the
intermediate roots get cached in memory in full while the computation is
ongoing - a future PR will need to address this deficiency, for example
by streaming the already-computed hashes directly to the database.
2024-07-04 16:51:50 +02:00
andri lim f04f30c72b
Reduce EVM complexity by removing forkOverride (#2448)
* Reduce EVM complexity by removing forkOverride

* Fixes
2024-07-04 15:48:36 +02:00
Jacek Sieka 81e75622cf
storage: store root id together with vid, for better locality of refe… (#2449)
The state and account MPT:s currenty share key space in the database
based on that vertex id:s are assigned essentially randomly, which means
that when two adjacent slot values from the same contract are accessed,
they might reside at large distance from each other.

Here, we prefix each vertex id by its root causing them to be sorted
together thus bringing all data belonging to a particular contract
closer together - the same effect also happens for the main state MPT
whose nodes now end up clustered together more tightly.

In the future, the prefix given to the storage keys can also be used to
perform range operations such as reading all the storage at once and/or
deleting an account with a batch operation.

Notably, parts of the API already supported this rooting concept while
parts didn't - this PR makes the API consistent by always working with a
root+vid.
2024-07-04 15:46:52 +02:00
andri lim b82dcdcc76
Remove unused StructLog (#2447) 2024-07-04 19:23:53 +07:00
andri lim d9e502bbc5
Bump web3/kzg4844/nimbus-eth2 and related fixes (#2446) 2024-07-04 05:41:32 +00:00
Jacek Sieka b23795ab39
remove pPrf, fRpp (#2445)
No longer used now that hashify is gone
2024-07-03 22:21:57 +02:00
Jacek Sieka 443c6d1f8e
Cache account path storage id (#2443)
The storage id is frequently accessed when executing contract code and
finding the path via the database requires several hops making the
process slow - here, we add a cache to keep the most recently used
account storage id:s in memory.

A possible future improvement would be to cache all account accesses so
that for example updating the balance doesn't cause several hikes.
2024-07-03 17:58:25 +02:00
tersec 989f20a740
bump nimbus-build-system for Nim v2.0.8 (#2440) 2024-07-03 17:58:02 +02:00
Jordan Hrycaj ea7c756a9d
Core db reorg (#2444)
* CoreDb: Merged all sub-descriptors into `base_desc` module

* Dissolve `aristo_db/common_desc.nim`

* No need to export `Aristo` methods in `CoreDb`

* Resolve/tighten methods in `aristo_db` sub-moduled

why:
  So they can be straihgt implemented into the `base` module

* Moved/re-implemented `KVT` methods into `base` module

* Moved/re-implemented `MPT` methods into `base` module

* Moved/re-implemented account methods into `base` module

* Moved/re-implemented `CTX` methods into `base` module

* Moved/re-implemented `handler_{aristo,kvt}` into `aristo_db` module

* Moved/re-implemented `TX` methods into `base` module

* Moved/re-implemented base methods into `base` module

* Replaced `toAristoSavedStateBlockNumber()` by proper base method

why:
  Was the last for keeping reason for keeping low level backend access
  methods

* Remove dedicated low level access to `Aristo` backend

why:
  Not needed anymore, for debugging the descriptors can be accessed
  directly

also:
  some clean up stuff

* Re-factor `CoreDb` descriptor layout and adjust base methods

* Moved/re-implemented iterators into `base_iterator*` modules

* Update docu
2024-07-03 15:50:27 +00:00
andri lim b1ff4ef3a5
Omit frame pointer for secp256k1 (#2442) 2024-07-03 10:14:03 +00:00
Jacek Sieka 1f60e8e453
Use `Hash256` directly for account path (#2439)
Account paths are always a hash - passing it around as such helps avoid
confusion as to how long it is
2024-07-03 10:14:26 +02:00
Jacek Sieka c364426422
Smaller in-database representations (#2436)
These representations use ~15-20% less data compared to the status quo,
mainly by removing redundant zeroes in the integer encodings - a
significant effect of this change is that the various rocksdb caches see
better efficiency since more items fit in the same amount of space.

* use RLP encoding for `VertexID` and `UInt256` wherever it appears
* pack `VertexRef`/`PayloadRef` more tightly
2024-07-02 20:25:06 +02:00
Kim De Mey 87d090afa6
Refactor history/beacon content types to seperate keys and values (#2438)
Refactor content types, to have similar structure as for state
network.
Amd some clean-up of comments/TODOs/spec links.
2024-07-02 15:48:49 +02:00
Kim De Mey d7b849db3d
Make portal_bridge cli options consistent and improve logging (#2437)
- Use --portal-rpc-url as url option to connect to Portal JSON-RPC
interface, just as --web3-url for EL JSON RPC interface.
- Improve logging to know beter which call on which JSON-RPC
interface fails
2024-07-02 11:24:41 +02:00
web3-developer e163b69261
Bump RocksDb version and enable autoClose on opt types to prevent memory leaks (#2427)
* Bump RocksDb version and enable autoClose on opt types to prevent memory leaks.
2024-07-02 13:44:09 +08:00
Jacek Sieka 3d3831dde8
Small cleanups (#2435)
* avoid costly hike memory allocations for operations that don't need to
re-traverse it
* avoid unnecessary state checks (which might trigger unwanted state
root computations)
* disable optimize-for-hits due to the MPT no longer being complete at
all times
2024-07-01 14:07:39 +02:00
Jordan Hrycaj 2c87fd1636
Aristo code cosmetics and tests update (#2434)
* Update some docu

* Resolve obsolete compile time option

why:
  Not optional anymore

* Update checks

why:
  The notion of what constitutes a valid `Aristo` db has changed due to
  (even more) lazy calculating Merkle hash keys.

* Disable redundant unit test for production
2024-07-01 10:59:18 +00:00
andri lim 740882d8ce
Import forked_chain_test in all_tests (#2433) 2024-07-01 09:57:42 +07:00
andri lim 401537ad38
Add ForkedChainRef tests (#2430)
ForkedChainRef have become quite complex.
test_blockchain_json is not sufficient cover for edge cases
or synthetic cases.
2024-06-30 14:40:14 +07:00
andri lim 403b93104b
Fix t8n: Calculate excessBlobGas from parent if not supplied (#2432)
* Fix t8n: Calculate excessBlobGas from parent if not supplied

* Fix comment
2024-06-30 14:39:45 +07:00
andri lim c24affadee
Use simpler schema when writing transactions, receipts, and withdrawals (#2420)
* Use simpler schema when writing transactions, receipts, and withdrawals

Using MPT not only slow but also take up more spaces than needed.
Aristo will remove older tries and only keep the last block tries.
Using simpler schema will avoid those problems.

* Rename getTransaction to getTransactionByIndex
2024-06-29 12:43:17 +07:00
Jacek Sieka 55ebd70d1e
stats: interpolate, remove some broken stats 2024-06-29 06:36:35 +02:00
andri lim b751d3adee
Combine smaller tests into bigger one (#2425)
1. test_state_db and test_ledger -> test_ledger.
   They are the same thing now.
2. stack, memory, code_stream, gas_meter, misc,
   overflow -> test_evm_support.
   They are small tests and fall into the same area.
2024-06-29 08:57:30 +07:00
Jordan Hrycaj 8dd038144b
Some cleanups (#2428)
* Remove `dirty` set from structural objects

why:
  Not used anymore, the tree is dirty by default.

* Rename `aristo_hashify` -> `aristo_compute`

* Remove cruft, update comments, cosmetics, etc.

* Simplify `SavedState` object

why:
  The key chaining have become obsolete after extra lazy hashing. There
  is some available space for a state hash to be maintained in future.

details:
  Accept the legacy `SavedState` object serialisation format for a
  while (which will be overwritten by new format.)
2024-06-28 18:43:04 +00:00
Jordan Hrycaj 14c3772545
On demand mpt revisited (#2426)
* rebased from `github/on-demand-mpt`

ackn:
  wip: on-demand mpt construction

  Given that actual data is stored in the `Vertex` structure, it's useful
  to think of the MPT as a cache for computing roots rather than being a
  functional requirement on its own.

  This PR engenders this line of thinking by incrementally computing the
  MPT only when it's needed, ie when a state (or similar) root is needed.

  This has the effect of siginficantly reducing memory usage as well as
  improving performance:

  * no need for dirty-mpt-node book-keeping
  * no need to build complex forest of upcoming hashing work
  * only hashes that are functionally needed are ever computed -
  intermediate nodes whose MTP root is not observed are never computed /
  processed

* Unit test hot fixes

* Unit test hot fixes cont.

(somehow lost that part)

---------

Co-authored-by: Jacek Sieka <jacek@status.im>
2024-06-28 15:03:12 +00:00