Commit Graph

601 Commits

Author SHA1 Message Date
Jordan Hrycaj 42a08cfba9
Coredb and sync maintenance update (#2583)
* bump metrics

* Remove cruft

* Cosmetics, update some logging, noise control

* Renamed `CoreDb` function `hasKey` => `hasKeyRc` and provided `hasKey`

why:
  Currently, `hasKey` returns a `Result[]` rather than a `bool` which
  is what one would expect from a function prototype of this name.

  This was a bit of an annoyance and cost unnecessary attention.
2024-08-30 11:18:36 +00:00
Jordan Hrycaj 4db9c5c2d5
Small updates and fixes for rlpx suite (#2571)
* Remove redundant `eth/68` message and clean up docu

details:
  There is only eth/68 available at the moment

* Allow to turn on chronicles line number logging in `Makefile`

* Accept (and forget) tx hashes announcements

why:
  Does no harm to just ignore it at the moment

* Bump nim-eth (rlp fix)
2024-08-19 14:00:10 +00:00
web3-developer c8d34eba9b
Bump portal-spec-tests and update Fluffy state tests (#2568)
* Bump portal-spec-tests.

* Update state network tests to use block_header instead of state_root.
2024-08-19 14:45:54 +08:00
web3-developer 9699293bfc
Update nim-rocksdb to latest version and cleanup outdated RocksDb install instructions in readme. (#2566) 2024-08-16 08:23:04 +02:00
Jordan Hrycaj d148de5b1c
Remove chunked rlpx (#2562)
* bump nim-eth

* Update make environment
2024-08-14 10:56:49 +00:00
Jacek Sieka ec7dd5ccf6
results: bump (lent) (#2561) 2024-08-14 11:02:14 +02:00
Jacek Sieka e3908a7b0d
results: bump (genericsOpenSym support) (#2559)
Makes `error` in `valueOr` behave.
2024-08-13 14:50:43 +02:00
Jacek Sieka b3184b716e
bncurve: bump (#2555)
...for significant speed gain
2024-08-09 13:02:30 +02:00
Jacek Sieka 32e2206d68
fix fluffy/eth (#2556)
* fix fluffy/eth

* revert speedups
2024-08-09 09:00:00 +02:00
Jacek Sieka 094486d0ce
Hash bump 2024-08-08 07:46:35 +02:00
web3-developer 93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
andri lim 77511b735e
Bump nim-libp2p to f8d4da6421b767727dd1501cb31b27636af87755 (#2534) 2024-07-28 14:26:23 +07:00
andri lim ddc8cbcdb9
Bump nim-metrics, nim-libbacktrace, nim-nat-traversal (#2533) 2024-07-27 18:46:35 +07:00
andri lim c3bb0e54c2
Bump json family submodules (#2532) 2024-07-27 18:46:21 +07:00
andri lim d20abfa5f2
Bump chronos and friends (#2531) 2024-07-27 18:46:07 +07:00
andri lim f26b70a5ab
Bump crypto submodules (#2530) 2024-07-27 18:45:54 +07:00
Kim De Mey bb18df0fac
Bump rocksdb for macOS rebuild fix (#2529) 2024-07-26 18:57:36 +02:00
Kim De Mey 84ce0c912e
Bump Portal test vectors and re-activate skipped tests (#2515) 2024-07-23 11:59:59 +02:00
web3-developer c0f5bd4e94
Update nim-rocksdb to latest. Fixes re-building issue on MacOS. (#2512) 2024-07-22 22:17:37 +08:00
andri lim 57b36dc4ee
Bump nim-stew to 54cc67cbb83f61b6e3168b09701758c5b805120a (#2505) 2024-07-19 16:23:54 +07:00
Kim De Mey 51cf991439
Bump ssz_serialization and use ByteList[n] + add ContentKeyByteList (#2500) 2024-07-17 17:07:27 +02:00
Kim De Mey 3bb707422b
Rework beacon block proofs to better structure (#2493)
The 3 proofs can be reworked to two proofs as we can use the
BeaconBlock directly instead of BeaconBlockHeader and
BeaconBlockBody. This is possible because the HTR of the
BeaconBlock is the same as the one of the BeaconBlockHeader.

This results in 32 bytes less as an intermediate hash can be
removed. But more importantly looks more clean and compact in
structure and code.
2024-07-17 11:32:05 +02:00
Kim De Mey 70682cd4a8
Bump NimYAML module and related changes (#2474) 2024-07-12 09:30:50 +02:00
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey 5ae5bd8b69
Bump portal-mainnet for updated fluffy bootstrap ENRs (#2469) 2024-07-10 15:45:11 +02:00
web3-developer 9fc5495d49
Update nim-rocksdb to latest version. (#2467) 2024-07-10 14:18:02 +08:00
Jacek Sieka 25b5f01357
bump stint (#2465)
avoids extreme modmul bottleneck
2024-07-09 18:07:21 +02:00
andri lim 4fa3756860
Convert GasInt to uint64, bump nim-eth and nimbus-eth2 (#2461)
* Convert GasInt to uint64, bump nim-eth and nimbus-eth2

* Bump nimbus-eth2

* int64.high.GasInt instead of 0x7fffffffffffffff.GasInt
2024-07-07 06:52:11 +00:00
tersec 1f40b710ee
fix UnusedImport warnings; bump nim-bearssl, nim-stint, and nim-stew (#2456) 2024-07-05 06:46:59 +07:00
andri lim d9e502bbc5
Bump web3/kzg4844/nimbus-eth2 and related fixes (#2446) 2024-07-04 05:41:32 +00:00
tersec 989f20a740
bump nimbus-build-system for Nim v2.0.8 (#2440) 2024-07-03 17:58:02 +02:00
web3-developer e163b69261
Bump RocksDb version and enable autoClose on opt types to prevent memory leaks (#2427)
* Bump RocksDb version and enable autoClose on opt types to prevent memory leaks.
2024-07-02 13:44:09 +08:00
Kim De Mey ae094692fb
Bump nim-eth for ENR changes and related clean-up (#2422) 2024-06-27 17:59:08 +02:00
web3-developer ea94e8a351
Use RocksDb column family handles instead of name strings. (#2418)
* Bump RocksDb to latest and update Nimbus database to pass column family handles to RocksDb API.

* Bump RocksDb version.
2024-06-27 16:51:43 +08:00
Jacek Sieka 768307d91d
Cache code and invalid jump destination tables (fixes #2268) (#2404)
It is common for many accounts to share the same code - at the database
level, code is stored by hash meaning only one copy exists per unique
program but when loaded in memory, a copy is made for each account.

Further, every time we execute the code, it must be scanned for invalid
jump destinations which slows down EVM exeuction.

Finally, the extcodesize call causes code to be loaded even if only the
size is needed.

This PR improves on all these points by introducing a shared
CodeBytesRef type whose code section is immutable and that can be shared
between accounts. Further, a dedicated `len` API call is added so that
the EXTCODESIZE opcode can operate without polluting the GC and code
cache, for cases where only the size is requested - rocksdb will in this
case cache the code itself in the row cache meaning that lookup of the
code itself remains fast when length is asked for first.

With 16k code entries, there's a 90% hit rate which goes up to 99%
during the 2.3M attack - the cache significantly lowers memory
consumption and execution time not only during this event but across the
board.
2024-06-21 09:44:10 +02:00
andri lim c8ae6b1a92
Bump ssz-serialization to b71ebc41c8e5027580be77a9707df1a64e6d9c8b (#2393) 2024-06-19 14:14:15 +02:00
Jacek Sieka 41cf81f80b
Fix dboptions init (#2391)
For the block cache to be shared between column families, the options
instance must be shared between the various column families being
created. This also ensures that there is only one source of truth for
configuration options instead of having two different sets depending on
how the tables were initialized.

This PR also removes the re-opening mechanism which can double startup
time - every time the database is opened, the log is replayed - a large
log file will take a long time to open.

Finally, several options got correclty implemented as column family
options, including an one that puts a hash index in the SST files.
2024-06-19 10:55:57 +02:00
Kim De Mey 4fd2ecddec
Bump nim-eth/web3/kzg4844/nimbus-eth2 and related fixes (#2392)
Bump nim-eth, which requires nimbus-eth2 bump, which requires
bumps of web3 and kzg4844 + related fixes to all those bumps.
2024-06-19 08:57:45 +07:00
andri lim 86877004e6
Bump nim-json-rpc to f40a8ba5c87748d89392f59d0c0f7cddba06cb60 (#2388) 2024-06-19 01:28:37 +00:00
Miran ea0d18424a
use Nim 2.0.6 (#2384)
* use Nim 2.0.6

* Fixes for nim 2.0.6

* Workaround nim 2.0 array indexing issue

* Remove excess gcsafe pragma

* Oops, fix recursive template

* Fix imports

* Fluffy nph linting

---------

Co-authored-by: jangko <jangko128@gmail.com>
Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-19 01:27:54 +00:00
web3-developer e3d14bd921
Fluffy portal testnet support (#2383)
* Bump portal-mainnet repo.

* Update command line arguments and parsing on startup.

* Read in angelfood bootstrap nodes and update Fluffy guide.

* Configure subnetwork protocol ids.
2024-06-18 15:32:57 +08:00
tersec e1bb65fdfa
rm PoW hash function and validation support (#2372) 2024-06-16 10:22:06 +07:00
andri lim a36aa95fab
Bump nim-bearssl for gcc-14 compatibility (#2366) 2024-06-16 10:20:42 +07:00
Jacek Sieka af34f90fe4
fix `max_total_wal_size` which should be set on the DB (#2363) 2024-06-16 02:11:30 +00:00
andri lim c5508b8dac
Bump nim-blscurve for gcc-14 compatibility (#2365)
* Bump nim-blscurve for gcc-14 compatibility

* Fix evm/blscurve.nim pointer works when using blscurve_abi
2024-06-15 17:34:07 +00:00
andri lim 4c458190e9
Bump nim-kzg4844 and nimbus-eth2 for gcc-14 compatibility (#2357) 2024-06-14 21:41:59 +07:00
andri lim ad57575958
Bump nim-serialization to 005ee90cb6aa563cdd690910455ea05f916ead3f (#2358) 2024-06-14 17:42:12 +07:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Jordan Hrycaj 5a5cc6295e
Triggered write event for kvt (#2351)
* bump rockdb

* Rename `KVT` objects related to filters according to `Aristo` naming

details:
  filter* => delta*
  roFilter => balancer

* Compulsory error handling if `persistent()` fails

* Add return code to `reCentre()`

why:
  Might eventually fail if re-centring is blocked. Some logic will be
  added in subsequent patch sets.

* Add column families from earlier session to rocksdb in opening procedure

why:
  All previously used CFs must be declared when re-opening an existing
  database.

* Update `init()` and add rocksdb `reinit()` methods for changing parameters

why:
  Opening a set column families (with different open options) must span
  at least the ones that are already on disk.

* Provide write-trigger-event interface into `Aristo` backend

why:
  This allows to save data from a guest application (think `KVT`) to
  get synced with the write cycle so the guest and `Aristo` save all
  atomically.

* Use `KVT` with new column family interface from `Aristo`

* Remove obsolete guest interface

* Implement `KVT` piggyback on `Aristo` backend

* CoreDb: Add separate `KVT`/`Aristo` backend mode for debugging

* Remove `rocks_db` import from `persist()` function

why:
  Some systems (i.p `fluffy` and friends) use the `Aristo` memory
  backend emulation and do not link against rocksdb when building the
  application. So this should fix that problem.
2024-06-13 18:15:11 +00:00
web3-developer 060c759b01
Use vendor RocksDb dll on Windows. (#2341)
* Bump rocksdb version and update Makefile.

* Update readme.
2024-06-13 10:10:53 +08:00