Commit Graph

3990 Commits

Author SHA1 Message Date
Jordan Hrycaj a25ea63dec
Revert lazy implementation (#2585) 2024-09-02 10:34:42 +00:00
Jacek Sieka 84a72c8658
Use zstd compression in bottommost layer (#2582)
Tested up to block ~14m, zstd uses ~12% less space which seems to result
in a small:ish (2-4%) performance improvement on block import speed -
this seems like a better baseline for more extensive testing in the
future.

Pre: 57383308 kb
Post: 50831236 kb
2024-08-30 17:32:13 +02:00
Jordan Hrycaj 42a08cfba9
Coredb and sync maintenance update (#2583)
* bump metrics

* Remove cruft

* Cosmetics, update some logging, noise control

* Renamed `CoreDb` function `hasKey` => `hasKeyRc` and provided `hasKey`

why:
  Currently, `hasKey` returns a `Result[]` rather than a `bool` which
  is what one would expect from a function prototype of this name.

  This was a bit of an annoyance and cost unnecessary attention.
2024-08-30 11:18:36 +00:00
web3-developer ee6a7e8259
Fluffy state endpoint improvements (#2580)
* Return default values when account, slot or code doesn't exist.

* Handle case when storage doesn't exist due to account not existing or being a non contract account.
2024-08-28 16:27:36 +08:00
Jacek Sieka 8857fccb44
create per-fork opcode dispatcher (#2579)
In the current VM opcode dispatcher, a two-level case statement is
generated that first matches the opcode and then uses another nested
case statement to select the actual implementation based on which fork
it is, causing the dispatcher to grow by `O(opcodes) * O(forks)`.

The fork does not change between instructions causing significant
inefficiency for this approach - not only because it repeats the fork
lookup but also because of code size bloat and missed optimizations.

A second source of inefficiency in dispatching is the tracer code which
in the vast majority of cases is disabled but nevertheless sees multiple
conditionals being evaluated for each instruction only to remain
disabled throughout exeuction.

This PR rewrites the opcode dispatcher macro to generate a separate
dispatcher for each fork and tracer setting and goes on to pick the
right one at the start of the computation.

This has many advantages:

* much smaller dispatcher
* easier to compile
* better inlining
* fewer pointlessly repeated instruction
* simplified macro (!)
* slow "low-compiler-memory" dispatcher code can be removed

Net block import improvement at about 4-6% depending on the contract -
synthetic EVM benchmnarks would show an even better result most likely.
2024-08-28 10:20:36 +02:00
web3-developer fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer 8bf581e72d
Cleanup unused exp_getProofsByBlockNumber endpoint (#2577)
* Cleanup unused exp_getProofsByBlockNumber endpoint.
2024-08-23 22:39:33 +08:00
web3-developer 507a9e71df
Fluffy state network fixes and improvements (#2576)
* Cleanup tests.

* Improve offer sort function in state bridge.

* Verify nibble prefix for leaf and extension nodes during lookup.
2024-08-23 15:46:23 +08:00
Jacek Sieka dbabe7e0a7
import: reduce stack usage (#2575)
Because EthBlock is quite large, the stack usage that results from the
multiple copies (temporary and not) present in the import command is
larger than it should be - this PR moves some of that data to a closure
environment allocated once per EthBlock - a larger restructuring of the
code is due but in the meantime, this simple change speeds up garbage
collection a little bit.
2024-08-22 10:06:45 +02:00
Jacek Sieka 22dacdd81c
t8n: enable reverse slot hash map (#2573)
turns out tracer still needs it
2024-08-20 15:23:24 +02:00
Jacek Sieka d72a73de8b
avoid digest when loading era block (#2572)
Computing the digest is unnecessary but takes a little bit of time -
remove computation and reduce mem usage slightly when loading era blocks
2024-08-20 15:23:14 +02:00
Jordan Hrycaj 4db9c5c2d5
Small updates and fixes for rlpx suite (#2571)
* Remove redundant `eth/68` message and clean up docu

details:
  There is only eth/68 available at the moment

* Allow to turn on chronicles line number logging in `Makefile`

* Accept (and forget) tx hashes announcements

why:
  Does no harm to just ignore it at the moment

* Bump nim-eth (rlp fix)
2024-08-19 14:00:10 +00:00
web3-developer 60c9b2c00d
Fluffy State Bridge: Fetch portal client nodeId on startup and sort offers by distance from nodeId (#2570)
* Fetch portal client nodeId on startup and sort offers by distance from nodeId.

* Fix logging.

* Improve error handling at startup.
2024-08-19 20:49:07 +08:00
Jacek Sieka 9826557184
fix make_states script dir 2024-08-19 10:16:32 +02:00
Jacek Sieka 226cdb7c68
avoid exceptions, tx copy (#2569)
* avoid some exceptions using the new compile-time strformat
* avoid copying the full transaction only to normalise 2 fields
(expensive)
2024-08-19 09:42:07 +02:00
Jacek Sieka 5941fef211
Avoid unnecessary layer copy (#2567)
When the stack has an empty layer on top, there's no need to copy the
contents of `top` to it since it would be the same.

~13% processing saved (!)

pre
```
INF 2024-08-17 19:11:31.748+02:00 Imported blocks
blockNumber=18667648 blocks=12000 importedSlot=7860043 txs=1797812
mgas=181135.177 bps=8.763 tps=1375.062 mgps=132.125 avgBps=6.798
avgTps=1018.501 avgMGps=102.617 elapsed=29m25s154ms
```

post
```
INF 2024-08-17 18:22:52.513+02:00 Imported blocks
blockNumber=18667648 blocks=12000 importedSlot=7860043 txs=1797812
mgas=181135.177 bps=9.648 tps=1513.961 mgps=145.472 avgBps=7.876
avgTps=1179.998 avgMGps=118.888 elapsed=25m23s572ms
```
2024-08-19 08:46:04 +02:00
web3-developer c8d34eba9b
Bump portal-spec-tests and update Fluffy state tests (#2568)
* Bump portal-spec-tests.

* Update state network tests to use block_header instead of state_root.
2024-08-19 14:45:54 +08:00
web3-developer 9699293bfc
Update nim-rocksdb to latest version and cleanup outdated RocksDb install instructions in readme. (#2566) 2024-08-16 08:23:04 +02:00
Jacek Sieka 43d93bcdab
Don't write slot hashes on import (#2564)
The reverse slot hash mechanism causes quite a bit of database traffic
but is broadly not useful except for iterating the storage of an
account, something that a validator never does (it's used by the
tracers).

This flag adds one more thing that is not stored in the database, to be
explored more comprehensively when designing full, validator and archive
modes with different pruning options in the future.

`ldb` says this is 60gb of data (!):
```
ldb --db=. --ignore_unknown_options --column_family=KvtGen approxsize
--hex --from=0x05
--to=0x05ffffffffffffffffffffffffffffffffffffffffffffff
66488353954
```
2024-08-16 08:22:51 +02:00
Jordan Hrycaj 4dbc1653ea
Cleanup (#2565)
* Move snap un-dumpers to aristo unit test folder

why:
  The only place where it is used, now to test the database against
  legacy snap sync dump samples.

   While the details of the dumped data have mostly outlived their purpuse,
   its use as **entropy** data thrown against `Aristo` has still been
   useful to find/debug tricky DB problems.

* Remove cruft

* `nimbus-eth1-blobs` not used anymore as test data source
2024-08-15 12:31:07 +00:00
Jordan Hrycaj cbe5131927
Simplify aristo tree deletion functionality (#2563)
* Cleaning up, removing cruft and debugging statements

* Make `aristo_delta` fluffy compatible

why:
  A sub-module that uses `chronicles` must import all possible
  modules used by a parent module that imports the sub-module.

* update TODO
2024-08-14 12:09:30 +00:00
Jordan Hrycaj d148de5b1c
Remove chunked rlpx (#2562)
* bump nim-eth

* Update make environment
2024-08-14 10:56:49 +00:00
Jacek Sieka ec7dd5ccf6
results: bump (lent) (#2561) 2024-08-14 11:02:14 +02:00
Jordan Hrycaj ce713d95fc
Aristo lazily delete larger subtrees (#2560)
* Extract sub-tree deletion functions into separate sub-modules

* Move/rename `aristo_desc.accLruSize` => `aristo_constants.ACC_LRU_SIZE`

* Lazily delete sub-trees

why:
  This gives some control of the memory used to keep the deleted vertices
  in the cached layers. For larger sub-trees, keys and vertices might be
  on the persistent backend to a large extend. This would pull an amount
  of extra information from the backend into the cached layer.

  For lazy deleting it is enough to remember sub-trees by a small set of
  (at most 16) sub-roots to be processed when storing persistent data.
  Marking the tree root deleted immediately allows to let most of the code
  base work as before.

* Comments and cosmetics

* No need to import all for `Aristo` here

* Kludge to make `chronicle` usage in sub-modules work with `fluffy`

why:
  That `fluffy` would not run with any logging in `core_deb` is a problem
  I have known for a while. Up to now, logging was only used for debugging.

  With the current `Aristo` PR, there are cases where logging might be
  wanted but this works only if `chronicles` runs without the
  `json[dynamic]` sinks.

  So this should be re-visited.

* More of a kludge
2024-08-14 08:54:44 +00:00
Jacek Sieka e3908a7b0d
results: bump (genericsOpenSym support) (#2559)
Makes `error` in `valueOr` behave.
2024-08-13 14:50:43 +02:00
Jordan Hrycaj 7becf4e389
Remove vertex ID recycle function (#2558)
why:
  It is not safe in general to recycle vertex IDs while the `RocksDb`
  cache has `VertexID` rather than `RootedVertexID` where the former
  type seems preferable.

  In some fringe cases one might remove a vertex with key `(root1,vid)`
  and insert another vertex with key `(root2,vid)` while re-using the
  vertex ID `vid`. Without knowledge of `root1` and `root2`, the LRU
  cache will return the same vertex for `(root2,vid)` also for
  `(root1,vid)`.
2024-08-12 20:56:15 +00:00
Jacek Sieka 8723a79225
add era dir to make_states 2024-08-12 14:49:32 +02:00
Jacek Sieka 19451cadff
rebalance rocksdb cache sizes (#2557)
Based on some simple testing done with a few combinations of cache
sizes, it seems that the block cache has grown in importance compared to
the where we were before changing on-disk format and adding a lot of
other point caches.

With these settings, there's roughly a 15% performance increase when
processing blocks in the 18M range over the status quo while memory
usage decreases by more than 1gb!

Only a few values were tested so there's certainly more to do here but
this change sets up a better baseline for any future optimizations.

In particular, since the initial defaults were chosen root vertex id:s
were introduced as key prefixes meaning that storage for each account
will be grouped together and thus it becomes more likely that a block
loaded from disk will be hit multiple times - this seems to give the
block cache an edge over the row cache, specially when traversing the
storage trie.
2024-08-12 05:52:09 +00:00
Jacek Sieka b3184b716e
bncurve: bump (#2555)
...for significant speed gain
2024-08-09 13:02:30 +02:00
Jacek Sieka 32e2206d68
fix fluffy/eth (#2556)
* fix fluffy/eth

* revert speedups
2024-08-09 09:00:00 +02:00
andri lim b8e128203f
Rewire blockValue from Txpool to EngineAPI (#2554) 2024-08-09 06:05:18 +07:00
Jacek Sieka 3dc30195ad
log http/jwt information on startup (#2553) 2024-08-08 10:03:30 +00:00
Jacek Sieka 094486d0ce
Hash bump 2024-08-08 07:46:35 +02:00
Jacek Sieka 3cefd7ed38
move db init to init (#2552)
When using the common interface, the database always (potentially) needs
init - take the opportunity to log some basic database info on startup.
2024-08-08 07:45:30 +02:00
web3-developer 93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
andri lim d5786758b5
TxPool: Merge tx_chain and tx_packer to reduce complexity (#2549)
* TxPool: Merge tx_chain and tx_packer to reduce complexity

* Fix copyright year
2024-08-07 22:35:17 +07:00
Jordan Hrycaj 38572bd8ea
Cache a storage root ID forever in the leaf payload of an account (#2551)
details:
  Stale root IDs are marked disabled while the ID is kept in the leaf
  payload.

why:
  This might lead to further caching advantages.
2024-08-07 13:28:01 +00:00
Jordan Hrycaj 488bdbc267
Provide portal proof functionality with coredb (#2550)
* Provide portal proof functions in `aristo_api`

why:
  So it can be fully supported by `CoreDb`

* Fix prototype in `kvt_api`

* Fix node constructor for account leafs with storage trees

* Provide simple path check based on portal proof functionality

* Provide portal proof functionality in `CoreDb`

* Update TODO list
2024-08-07 11:30:55 +00:00
andri lim 3cef119b78
Return empty list instead of error in getPooledTxs handler (#2547) 2024-08-06 22:06:48 +07:00
Jordan Hrycaj 6bae929439
Added comments (#2546) 2024-08-06 12:43:39 +00:00
Jordan Hrycaj 5b502a06c4
Added portal proof nodes generation functionality (#2539)
* Extracted `test_tx.testTxMergeProofAndKvpList()` => separate file

* Fix serialiser

why:
  Typo lead to duplicate rlp-encoded nodes in chain

* Remove cruft

* Implemnt portal proof nodes generators `partXxxTwig()`

* Add unit test for portal proof nodes generator `partAccountTwig()`

* Cosmetics

* Simplify serialiser return code format

* Fix proof generator for extension nodes

why:
  Code was simply bonkers, not detected before the unit tests were
  adapted to check for just this.

* Implemented portal proof nodes verifier `partUntwig()`

* Cosmetics

* Fix `testutp` cli poblem
2024-08-06 11:29:26 +00:00
andri lim ec118a438a
Refactor txpool: reduce complexity (#2542) 2024-08-06 16:12:56 +07:00
web3-developer 63d13182c1
Fluffy state bridge - Add CLI parameters and implement workers for offer gossip. (#2541)
* Use RPC batching to send offer requests and filter out duplicate offers.

* Lookup offers after gossip to check if gossip successful.

* Use multiple workers for gossiping offers.

* Update Fluffy state network logging.

* Use single RPC calls instead of batching.

* Update cli parameters.

* Fix bug in contract trie offer building.
2024-08-06 15:38:38 +08:00
andri lim 9dacfed943
Disable txpool in eth wire protocol handler (#2540) 2024-08-06 11:26:55 +07:00
Jordan Hrycaj 01b5c08763
Revive json tracer unit tests (#2538)
* Some `Aristo` clean-ups/updates

* Re-implemented core-db tracer functionality

* Rename nimbus tracer `no-tracer.nim` => `tracer.nim`

why:
  Restore original name for easy diff tracking with upcoming update

* Update nimbus tracer using new core-db tracer functionality

* Updating json tracer unit tests

* Enable json tracer unit tests
2024-08-01 10:41:20 +00:00
andri lim e331c9e9b7
TxPool: Replace GasPrice and GasPriceEx with GasInt (#2537)
* TxPool: Replace GasPrice and GasPriceEx with GasInt
2024-07-31 14:33:30 +07:00
web3-developer 947f629903
Fluffy State Bridge - State Gossip via Portal JSON-RPC (#2535)
* Create block offers queue and collect account preimages.

* Implement iterators to return account and storage proofs and bytecode from updatedCaches.

* Implement building offers from proofs.

* Refactor BlockDataRef type to only include required fields.

* Store block data in database.

* Improve state diff types.

* Implement start state backfill from specific block.

* Record last persisted block number in database.

* Persist account preimages in db.

* Apply state updates for DAO hard fork.

* Implement state gossip of block offers via portal JSON RPC.
2024-07-30 22:56:21 +08:00
Jordan Hrycaj 72c3ab8ced
Provide partial tree support for preloading tests (#2536)
* Implement partial trees

why:
  This is currently needed for unit tests to pre-load the database
  with test data similar to `proof` node pre-load.

  The basic features for `snap-sync` boundary proofs are available
  as well for future use. What is missing is the final proof verification
  and a complete storage data load/merge function (stub is available.)

* Cosmetics, clean up
2024-07-29 20:15:17 +00:00
andri lim 77511b735e
Bump nim-libp2p to f8d4da6421b767727dd1501cb31b27636af87755 (#2534) 2024-07-28 14:26:23 +07:00
andri lim ddc8cbcdb9
Bump nim-metrics, nim-libbacktrace, nim-nat-traversal (#2533) 2024-07-27 18:46:35 +07:00