Commit Graph

3826 Commits

Author SHA1 Message Date
Jordan Hrycaj 99238ce0e4
Core db maintenance update (#2087)
* CoreDb+Aristo: Fix handler code

* Aristo+Kvt: Remove cruft

* Aristo+Kvt: The function `forkTop()` always provides a single transaction

why:
  Previously it provided a single squashed tx only if there were any. Now
  it will provide a blind one if there were none.

* Fix Copyright header
2024-03-20 15:15:56 +00:00
andri lim c41206be39
Fix styles and reduce compiler warnings (#2086)
* Fix styles and reduce compiler warnings

* Fix copyright year
2024-03-20 14:35:38 +07:00
Kim De Mey 49a199dbc3
Add beacon state historical_roots to fluffy (#2085)
- Add historical_roots in the binary for proof verification
pre-capella
- Add command to export historical_roots to file in
eth_data_exporter tool
2024-03-19 16:45:32 +01:00
web3-developer a147ff7553
Fix 'value out of range' RangeDefect in block witness generation. (#2072)
* Fix 'value out of range' RangeDefect caused by large/expensive blocks/transactions during DOS period.

* Clear witness cache in AccountCache persist.

* Revert previous fix and force clear cache after processing each block.

* Revert clear cache in process block.
2024-03-19 15:41:40 +08:00
Jordan Hrycaj 14a5f46d13
Core db implement ctx layer for mpt state admin (#2082)
* CoreDb+Ledger: Update logging

why:
  Use symbol `api` rather than `ctx` because the latter will be used
  as name for particular objects

* CoreDb: Remove cruft

* CoreDb: Remove `TxID` support

why:
  It is nowhere used and ugly implemented. The upcoming context layer
  will be a cleaner alternative to use, instead should this particular
  functionality be needed.

* CoreDb: Rearrange base methods in source code for better reading

* CoreDb+Aristo: Update API closures for better reading & maintenance

* CoreDb: Implement context layer for MPT

why:
  On `Aristo` the context layer allows to manage different views on
  the same backend database. This is an abstraction of the legacy
  hexary trie which can be localised on a particular root nose.

details:
  The `ctx` context provides the state (equiv. to state root) of the
  database for MPT and account descriptors.

* Fix Copyright headers
2024-03-18 19:40:23 +00:00
Jordan Hrycaj 5379302ce9
Aristo+Kvt: Let destructor crash when `nil` argument is given (#2080)
why:
  Ignoring `nil` objects was handy for a while but eventually led to
  lazy programming which in turn led to double destructor calls for
  the rocks-db.
2024-03-15 14:20:00 +00:00
andri lim 90622c0915
Bump libtommath to f96ffdf38803218fe74079738c9ff888bc9ab48c (#2069) 2024-03-15 20:52:41 +07:00
Jordan Hrycaj 1da58e8244
CoreDb+Aristo: Fix missing reference (#2079) 2024-03-15 12:35:35 +00:00
Kim De Mey bfa756c36b
Bump json-rpc and web3 and remove unneeded errors import/export (#2078) 2024-03-15 11:44:18 +01:00
Kim De Mey 461c7e7d85
Add Era1 backfill to portal_bridge history mode (#2077) 2024-03-15 10:41:41 +01:00
Jordan Hrycaj 0d73637f14
Core db simplify new api storage modes (#2075)
* Aristo+Kvt: Fix backend `dup()` function in api setup

why:
  Backend object is subject to an inheritance cascade which was not
  taken care of, before. Only the base object was duplicated.

* Kvt: Simplify DB clone/peers management

* Aristo: Simplify DB clone/peers management

* Aristo: Adjust unit test for working with memory DB only

why:
  This currently causes some memory corruption persumably in the
  `libc` background layer.

* CoredDb+Kvt: Simplify API for KVT

why:
  Simplified storage models (was over engineered) for better performance
  and code maintenance.

* CoredDb+Aristo: Simplify API for `Aristo`

why:
  Only single database state needed here. Accessing a similar state will
  be implemented from outside this module using a context layer. This
  gives better performance and improves code maintenance.

* Fix Copyright headers

* CoreDb: Turn off API tracking

why:
  CI would ot go through. Was accidentally turned on.
2024-03-14 22:17:43 +00:00
Kim De Mey 0fb7632aaa
Fix BlockNumber conversion for getBlockReceipts in portal_bridge (#2074) 2024-03-14 17:21:10 +01:00
Kim De Mey 7287efc7b4
Clean-up of some Portal json-rpc code (#2073)
- Use the new createRpcSigsFromNim for client json-rpc API
- Avoid importing any nimbus/rpc specifics, use only web3 and
fluffy local rpc code
- Adjust tools making use of the client side API
2024-03-13 16:58:50 +01:00
Kim De Mey 849607cfd1
Remove option of using beacon_lc_bridge together with a full node (#2070)
Keep the beacon_lc_bridge a fully standalone bridge, and instead
use the new portal_bridge which works by means of full nodes or
web3 providers.
2024-03-12 14:06:22 +01:00
web3-developer 799acf301d
Added support for namespaces to RocksDb kvstore. (#2066)
* Add new RocksNamespaceRef type and remove backups and readonly support from RocksDb KvStore.

* Bump nim-rocksdb to fc2ba4a836b6b47ae1b17d1c45801c7e06585e19

* Fix tests.

* Fix copyright notice.
2024-03-12 11:04:46 +08:00
Kim De Mey 1159b0114e
Add portal_bridge history mode (#2067)
Portal bridge mode for following latest and injecting the latest
block data into the Portal network.
2024-03-11 18:20:29 +01:00
Jordan Hrycaj 332ec75d5a
Database architecture diagram & module overview (#2065) 2024-03-08 18:42:46 +00:00
Kim De Mey f8fca07d37
Update protocol-interop-testing.md with correct dcli build dir (#2064) 2024-03-08 10:32:53 +01:00
Jordan Hrycaj 3e1e493368
Core n lega db update tracer api (#2063)
* Aristo: Remove cruft

* Prettifying profile statistics printing & source code cosmetics

* Aristo/Kvt: API tools update

* CoreDb: Corrections, mostly API related

* CoreDb: Changed the tracer `logDb()` result

why:
  Only the key-value table is used from the logger database

* CoreDb: Update legacy tracer

why:
  The `contains()` directive did not correspond to `0 < get().len`

also:
  Copy `CoreDb` meta settings like `trackLegaApi`, `trackNewApi`, etc. to
  overlay tracer descriptor

* CoreDb: Extend/update tracer API

why:
  Get ready for accommodating `Aristo` tracer

* Fix missing import

why:
  Some CI compilers might have (cached?) a different NIM patch level

* Ditto
2024-03-07 19:24:05 +00:00
Kim De Mey 88a93beb26
Provide a specific error in JSON-RPC API when content is not found (#2059) 2024-03-06 08:57:09 +01:00
Kim De Mey 0debf1a122
Introduce --networks and --portal-network cli options (#2061)
The --networks option is to select the networks alias the Portal
sub-protocols.
The --network option is deprecated and replaced with the
--portal-network option to avoid confusion with --networks option.

This commit also remove the --state flag which was a temporary
quick-fix.
2024-03-05 13:45:48 +01:00
web3-developer 11691c33e9
Update Nimbus codebase to use the new nim-rocksdb API. (#2054)
* Bump nim-rocksdb.

* Update codebase to use latest nim-rocksdb API.

* Update copyright notices.

* Fix memory leak due to allocCStringArray without deallocCStringArray.

* Improve kvstore_rocksdb code.

* Refactor and cleanup RocksStoreRef.

* Update nim-rocksdb submodule to latest.
2024-03-05 12:54:42 +08:00
Jordan Hrycaj 587ca3abbe
Coredb use stackable api for aristo backend (#2060)
* Aristo/Kvt: Provide function hooks APIs

why:
  These APIs can be used for installing tracers, profiling functoinality,
  and other niceties on the databases.

* Aristo: Provide optional API profiling

details:
  It basically is a re-implementation of the `CoreDb` profiling
  implementation

* Kvt: Provide optional API profiling similar to `Aristo`

* CoreDb: Re-implementing profiling using `aristo_profile`

* Ledger: Re-implementing profiling using `aristo_profile`

* CoreDb: Update unit tests for maintainability

* update copyright dates
2024-02-29 21:10:24 +00:00
Kim De Mey 7089226d43
Formatting nimbus_verified_proxy with nph v0.5.1 (#2058)
* Adjust CI linter nph version to v0.5.1

* Formatting nimbus_verified_proxy with nph v0.5.1
2024-02-29 18:56:59 +01:00
andri lim 7079c7b305
Bump nim-web3 to 747001250a27278e0f0f1706fe1dec88ad3ede8d (#2057)
Remove error field from AccessListResult
Add optional yParity field to TransactionObject
2024-02-29 22:02:38 +07:00
andri lim a32a6200b6
Bump json-serialization to 7516a92eb8339f8fc1de7dd16cfa3d77bce87cd2 (#2056)
- avoid XDeclaredButNotUsed when no field isnot JsonVoid (#87)
- Fix JsonValueRef.Object comparison
2024-02-29 20:16:36 +07:00
andri lim 7722a907dc
Implement getAccessList of db/ledger (#2055) 2024-02-29 18:16:47 +07:00
andri lim 66c0f92cc6
Bump nim-stint to 7bb0ccc7842577243edc3b8d61f75bf498551a7a (#2053)
Make decimal string parser raise ValueError instead of RangeDefect. (#148)
Upgrade github actions to v4 (#149)
2024-02-29 07:38:31 +07:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
Kim De Mey b656875778
Quickfix for possible IndexError in state network recursive gossip (#2052)
This would currently manifest sometimes in state gossip test in
CI or locally. But could occur on fluffy binary in case state
network  is enable and data is gossiped.
2024-02-27 17:39:23 +01:00
andri lim d830692b72
Bump nim-evmc: add push raises: [] (#2051) 2024-02-24 09:38:50 +07:00
Jordan Hrycaj a02a915039
Provide public default db symbol (#2050)
* CoreDb: Provide default db backend symbols

why:
  Handy for running `Aristo` against standard tests

note:
  These defaults are currently set to legacy DB types. The must be
  enabled manually in `db/core_db.nim`.

* Provide `Aristo` for macro assembler related tests

caveat:
  Some tests use `initStorageTrie()` which lets `Aristo` bail out. The
  test need to run on `distinct_ledgers` (or something like) rather than
  `distinct_tries`.

* Tests: Misc modules that can run on `Aristo` as well

* NoHive: Module that can run on `Aristo` as well

* Fix copyright year

* ditto
2024-02-23 09:17:24 +00:00
Jordan Hrycaj 8e18e85288
Aristodb remove obsolete and time consuming admin features (#2048)
* Aristo: Reorg `hashify()` using different schedule algorithm

why:
  Directly calculating the search tree top down from the roots turns
  out to be faster than using the cached structures left over by `merge()`
  and `delete()`.
  Time gains is short of 20%

* Aristo: Remove `lTab[]` leaf entry object type

why:
  Not used anymore. It was previously needed to build the schedule for
  `hashify()`.

* Aristo: Avoid unnecessary re-org of the vertex ID recycling list

why:
  This list can become quite large so a heuristic is employed whether
  it makes sense to re-org.

  Also, re-org check is only done by `delete()` functions.

* Aristo: Remove key/reverse lookup table from tx layers

why:
  It is ignored except for handling proof nodes and costs unnecessary
  run time resources.

  This feature was originally needed to accommodate the mental transition
  from the legacy MPT to the `Aristo` trie :).

* Fix copyright year
2024-02-22 08:24:58 +00:00
andri lim 3a1b0d8674
Bump json-serialization and nimbus-eth2 (#2049)
* Bump json-serialization and nimbus-eth2

* Remove deprecated ValidIpAddress usage
2024-02-22 12:15:25 +07:00
andri lim f2694d86e0
Bump submodules (#2045) 2024-02-22 09:03:15 +07:00
andri lim 6ff2edc416
Fix styles (#2046)
* Fix styles

* Fix copyright year
2024-02-21 23:04:59 +07:00
andri lim 3a4da2cdb7
Bump eth_tests to 5bf1fff257c0662bc526c27d7546d470025b3239 (#2047) 2024-02-21 23:04:44 +07:00
andri lim bd8412a35e
use non-EOL macOS version for GitHub Actions CI (#2044) 2024-02-21 16:15:05 +07:00
andri lim 52aa87c539
Align with EIP-4844: rename getBlobGasPrice to getBlobBaseFee (#2043) 2024-02-21 16:14:34 +07:00
andri lim 7b4ef814ea
Bump eth_tests and related fixes (#2042)
* Bump eth_tests and related fixes

* Fix tests
2024-02-21 16:14:20 +07:00
andri lim 4735af8603
Bump cryptographics libraries (#2041) 2024-02-21 16:14:00 +07:00
Daniel Sobol 95a9145f9e
add validators for data received from state network via contentLookup (#2039) 2024-02-20 16:15:24 +03:00
andri lim b09fba0b49
Refactor EVM raises to reduce compiler warnings (#2040)
* Fix style check when EVMC enabled

* Refactor EVM raises to reduce compiler warnings

* Fix EVM handler raises when EVMC enabled

* Workaround stupid style checker false complaints

* Fix Windows/clang linking error due to function pointer incompatibility
2024-02-20 14:16:12 +07:00
andri lim 7c1af9a78f
Add style check to config.nims and fix styles in source code (#2038)
* Add style check to config.nims and fix styles in source code

* Fix copyright year
2024-02-20 10:07:38 +07:00
Daniel Sobol 47b254d8b5
State network content retrieval (#2033)
* initial rework of contentLoop for state network

* prepare content for storage and retrieval

* modify test to account for retrieval data
2024-02-19 17:35:53 +03:00
andri lim 402a3eda73
Refactor protocol handlers exception handling (#2037)
* Refactor protocol handlers exception handling

* Fix copyright year
2024-02-19 18:46:48 +07:00
andri lim 2a1386e46a
Bump nim-eth: Fix improper yield usage (#2036) 2024-02-19 16:33:55 +07:00
andri lim 795d0d779e
Bump json-rpc to 9c65df305390ab01a13788af5aea78cd62dceb2a (#2035) 2024-02-19 16:33:43 +07:00
andri lim 5aa7a3b9f5
Remove nimble.lock (#2034) 2024-02-19 13:54:51 +07:00
Daniel Sobol 2766e20179
Add flag to explicitly enable state network (#2031) 2024-02-17 01:38:58 +03:00