Commit Graph

3766 Commits

Author SHA1 Message Date
Kim De Mey 95089f103a
Bump portal-spec-tests for additional header validation tests (#2208) 2024-05-22 19:23:18 +02:00
Kim De Mey ba2dd76f30
Rename the Portal testnet0 to mainnet (#2204)
Also adds the new testnet as option, but this needs to be still
implemented by adding testnet bootstrap nodes and protocol ids.
2024-05-22 17:10:31 +02:00
Jordan Hrycaj 9cc6e5a3aa
Aristo resume off line syncing on pre loaded database (#2203)
* Update some docu & messages

* Remove cruft from the ledger modules

* Must not overwrite genesis data on an initialised database

why:
  This will overwrite the global state of the Aristo single state DB.
  Otherwise resuming at the last synced state becomes impossible.

* Provide latest block number from journal

why:
  This relates the global state of the DB directly to the corresponding
  block number.

* Implemented unit test providing DB pre-load and resume
2024-05-22 13:41:14 +00:00
Jacek Sieka 308bd51bbd
era: speed up index reading (#2202) 2024-05-22 09:27:23 +02:00
web3-developer e566e89d03
Fluffy state network refactor (#2200)
* Add additional getTrieProof test.

* Refactor state network: Removed variant objects for values and split state_content into multiple files.
2024-05-22 09:22:07 +08:00
Jordan Hrycaj 2629d412d7
CoreDb+Aristo: Update tracer (#2201)
why:
  When deleting accounts while restoring the previous state, storage tries
  must be deleted first. Otherwise a `DelDanglingStoTrie` error will occur
  when trying to delete an account which refers to an active storage trie.
2024-05-21 12:51:06 +00:00
Jordan Hrycaj bc8e3c02ee
Fix missing disabling of `eth` service (#2199)
why:
  Otherwise it might crash on a not fully initialised call back function
  when most other functions are disabled during the early sync state.
2024-05-20 15:02:36 +00:00
Jordan Hrycaj de0388919f
Unified mode for undumping gzip-ed or era1-ed encoded block dumps (#2198)
ackn:
  Built on Daniel's work
2024-05-20 13:59:18 +00:00
Jordan Hrycaj ee9aea171d
Culling legacy DB and accounts cache (#2197)
details:
+ Compiles nimbus all_tests
+ Failing tests have been commented out
2024-05-20 10:17:51 +00:00
andri lim 38eaebc5c7
TxPool: rank calculator now include profit from blob fee (#2196)
* TxPool: rank calculator now include profit from blob fee

* Fix copyright year
2024-05-20 14:30:48 +07:00
jangko 053fc79a8b
Engine-API simulator: allow testee client to import invalid block 2024-05-19 10:08:05 +07:00
web3-developer 047f8ec4b2
Bump portal spec tests and update state validation tests to use state roots in test vectors. (#2193)
* Bump portal spec tests to version a549242

* Update state validation tests to use state roots in portal spec test vectors.
2024-05-17 23:20:57 +08:00
andri lim 33ac53217f
Check invalid ancestor in engine-api and beacon-sync (#2192) 2024-05-17 08:38:46 +07:00
Jacek Sieka 293ce28e4d
remove geth_db (#2189)
doesn't look like it's used
2024-05-16 22:00:20 +07:00
Kim De Mey f1d7d2d62b
Support multiple test vectors to be run for HeaderWithProof tests (#2191) 2024-05-16 13:37:56 +02:00
web3-developer 2891b9aa7d
Fluffy state offer validation (#2170)
* Implemented offer validation interface.

* Implement high level offer validation steps.

* Completed state validation tests.

* Update validation to use result type.

* Update state proof verification tests to test offer verification.

* Query history network to get state root by block hash.

* Fix state network test and remove usage of CoreDb.

* Fix state network gossip test and PR comment updates.

* Add trieproof state validation tests and fix for short nodes.
2024-05-16 16:16:16 +08:00
andri lim 8767bbd10a
Fix engine simulator and improve logging (#2188)
* Fix engine simulator and improve logging

* Fix engine simulator genesis loader
2024-05-15 23:22:03 +07:00
andri lim 7a13127be3
Let OS choose port for RPC server in test_rpc (#2187)
To avoid address/port collision and prevent
"Failed to create server .. (48) Address already in use" error
when executing tests in parallel.
2024-05-15 13:45:44 +07:00
Etan Kissling c4c37302b1
Introduce wrapper type for EIP-4844 transactions (#2177)
* Introduce wrapper type for EIP-4844 transactions

EIP-4844 blob sidecars are a concept that only exists in the mempool.
After inclusion of a transaction into an execution block, only the
versioned hash within the transaction remains. To improve type safety,
replace the `Transaction.networkPayload` member with a wrapper type
`PooledTransaction` that is used in contexts where blob sidecars exist.

* Bump nimbus-eth2 to 87605d08a7f9cfc3b223bd32143e93a6cdf351ac

* IPv6 'listen-address' in `nimbus_verified_proxy`

* Bump nim-libp2p to 21cbe3a91a70811522554e89e6a791172cebfef2

* Fix beacon_lc_bridge payload conversion and conf.listenAddress type

* Change nimbus_verified_proxy.asExecutionData param to SomeExecutionPayload

* Rerun nph to fix asExecutionData style format

* nimbus_verified_proxy listenAddress

* Use PooledTransaction in nimbus-eth1 tests

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-05-15 10:07:59 +07:00
Etan Kissling 766823cd49
Bump nim-chronos to 8a306763cec8105fa83574b56734b0f66823f844 (#2183)
Co-authored-by: jangko <jangko128@gmail.com>
2024-05-14 12:58:32 +07:00
jangko 37e7e52944
Bump nim-graphql to 54dcec7f6f01a2f1e0d5115b7352a1e16182a655 2024-05-14 09:23:09 +07:00
Kim De Mey 2e01878afe
Add the epoch to the HistoricalSummariesKey (#2179) 2024-05-13 18:49:21 +02:00
Etan Kissling 1b8f2b0ea5
Workaround for missing implementation of 'pending' tag (#2178)
The transaction spammer from Kurtosis requests the nonce value of its
account based on 'pending' tag. As we lack that implementation, go with
the next best answer from the 'latest' tag. This does not include info
about transactions from the mempool that have not yet been executed.
2024-05-13 16:42:28 +03:00
Kim De Mey 56e478c701
Write beacon LC content test vectors to yaml instead of json (#2176) 2024-05-13 12:34:14 +02:00
jangko 47a86f0df6
Bump nim-web3 to de87f860874be944cdc3dfd08765c687fff736c4 2024-05-13 10:07:48 +07:00
Kim De Mey 3be298f58f
Add tests for beacon block proofs Capella and onwards (#2161) 2024-05-10 22:51:24 +02:00
andri lim d29c9fae17
Update t8n test vectors after EIP-7610 update (#2169)
* Skip call family gas cost overflow check if refund is negative

* Update t8n test vectors after EIP-7610 update
2024-05-08 09:27:37 +07:00
Jordan Hrycaj 54f784bef1
Kvt remodel tx and forked descriptors (#2168)
* Aristo: Generalise alien/guest interface for piggiback on database

* Aristo: Code cosmetics

* CoreDb+Kvt: Update transaction API

why:
  Use single addressable function `forkTx(backLevel: int)` as used
  in `Aristo`. So `Kvt` can be synced simultaneously to `Aristo`.

also:
  Refactored `kvt_tx.nim` in a similar fashion to `Aristo`.

* Kvt: Replace `LayerDelta` object by reference

why:
  Will be needed when introducing filters

* Kvt: Remodel backend filter facility similar to `Aristo`

why:
  This allows to operate on several KVT instances simultaneously.

* CoreDb+Kvt: Fix on-disk storage

why:
  Overlooked name change: `stow()` => `persist()` for permanent storage

* Fix copyright headers
2024-05-07 19:59:27 +00:00
jangko d1f21f39c1
Bump nim-kzg4844 to bd372ec8e673c928ed2ad5ca887b696aab58b0b3 2024-05-07 10:28:40 +07:00
jangko d261484dd7
evm: Reject contract creation if the storage is non-empty(EIP-7610) 2024-05-07 09:19:59 +07:00
web3-developer 5c4358a969
Add retrieval content validation to state network. (#2166)
* Add retrieval content validation to state network.

* Complete state validation tests for retrieval content.

* Improve naming functions and parameters.
2024-05-06 23:13:35 +08:00
Jordan Hrycaj 143f2e99f5
Core db+aristo fixes and tx handling updates (#2164)
* Aristo: Rename journal related sources and functions

why:
  Previously, the naming was hinged on the phrases `fifo`, `filter` etc.
  which reflect the inner workings of cascaded filters. This was
  unfortunate for reading/understanding the source code for actions where
  the focus is the journal as a whole.

* Aristo: Fix buffer overflow (path length truncating error)

* Aristo: Tighten `hikeUp()` stop check, update error code

why:
  Detect dangling vertex links. These are legit with `snap` sync
  processing but not with regular processing.

* Aristo: Raise assert in regular mode `merge()` at a dangling link/edge

why:
  With `snap` sync processing, partial trees are ok and can be amended.
  Not so in regular mode.

  Previously there was only a debug message when a non-legit dangling edge
  was encountered.

* Aristo: Make sure that vertices are copied before modification

why:
  Otherwise vertices from lower layers might also be modified

* Aristo: Fix relaxed mode for validity checker `check()`

* Remove cruft

* Aristo: Update API for transaction handling

details:
+ Split `aristo_tx.nim` into sub-modules
+ Split `forkWith()` into `findTx()` + `forkTx()`
+ Removed `forkTop()`, `forkBase()` (now superseded by new `forkTx()`)

* CoreDb+Aristo: Fix initialiser (missing methods)
2024-05-03 17:38:17 +00:00
web3-developer 7da36bf459
Minor refactor to packNibbles and unpackNibbles and add additional tests. (#2162) 2024-05-03 22:55:51 +08:00
Kim De Mey 24b4d52dea
Cleanup tags from yaml output and use 0xHex (#2160) 2024-05-03 16:27:13 +02:00
Kim De Mey df2dcf593c
Remove few more unneeded nimbus imports in Fluffy tests (#2159) 2024-05-03 12:31:54 +02:00
Kim De Mey 9c64b44c89
Add post Bellatrix beacon block proof exports (#2158)
- Add post Bellatrix Beacon block proof exporting to
eth_data_exporter
- Rework the Bellatrix Beacon block proof exports to us yaml
format
- Move some common code of the two different beacon block proofs
to beacon_chain_block_proof_common module
- Move + add to yaml utils to be usable not just for the tests but
also for eth data exporter
2024-05-03 11:50:18 +02:00
Jordan Hrycaj 961f63358e
Core db+aristo update recovery journal management (#2156)
* Aristo: Allow to define/set `FilterID` for journal filter records

why:
  After some changes, the `FilterID` is isomorphic to the `BlockNumber`
  scalar (well, the first 2^64 entries of a `BlockNumber`.)

  The needed change for `FilterID` is that the `FilterID(0)` value is
  valid part of the `FilterID` scalar. A non-valid `FilterID` entry is
  represented by `none(FilterID)`.

* Aristo: Split off function `persist()` as persistent version of `stow()`

why:
  In production, `stow(persistent=false,..)` is currently unused. So,
  using `persist()` rather than `stow(persistent=true,..)` improves
  readability and is better to maintain.

* CoreDb+Aristo: Store block numbers in journal records

why:
  This makes journal records searchable by block numbers

* Aristo: Rename some journal related functions

why:
  The name *journal* is more appropriate to api functions than something
   with *fifo* or *filter*.

* CoreDb+Aristo: Update last/oldest journal state retrieval

* CoreDb+Aristo: Register block number with state root in journal

why:
  No need anymore for extra lookup table `stRootToBlockNum` which maps
  a storage root -> block number.

* Aristo: Remove unused function `getFilUbe()` from api

* CoreDb: Remove now unused virtual table `stRootToBlockNum`

why:
  Was used to map a state root to a block number. This functionality
  is now embedded into the recovery journal backend.

* Turn of API tracking (will fail on `fluffy`)
2024-04-29 20:17:17 +00:00
Jordan Hrycaj 0d4ef023ed
Update aristo journal functionality (#2155)
* Aristo: Code cosmetics, e.g. update some CamelCase names

* CoreDb+Aristo: Provide oldest known state root implied

details:
  The Aristo journal allows to recover earlier but not all state roots.

* Aristo: Fix journal backward index operator, e.g. `[^1]`

* Aristo: Fix journal updater

why:
  The `fifosStore()` store function slightly misinterpreted the update
  instructions when translation is to database `put()` functions. The
  effect was that the journal was ever growing due to stale entries which
  were never deleted.

* CoreDb+Aristo: Provide utils for purging stale data from the KVT

details:
  See earlier patch, not all state roots are available. This patch
  provides a mapping from some state root to a block number and allows to
  remove all KVT data related to a particular block number

* Aristo+Kvt: Implement a clean up schedule for expired data in KVT

why:
  For a single state ledger like `Aristo`, there is only a limited
  backlog of states. So KVT data (i.e. headers etc.) are cleaned up
  regularly

* Fix copyright year
2024-04-26 13:43:52 +00:00
Kim De Mey 1512f95067
Remove unneeded nimbus imports (#2154)
Also specifically select from the still required
nimbus imports.
2024-04-25 15:21:09 +02:00
web3-developer d12300ecc9
Remove usages of CoreDb in Fluffy. (#2152)
* Remove references to CoreDb in Fluffy code.

* Add missing import.

* Remove coredb imports and use dynamic server port in tests.

* Fix formatting using nph.
2024-04-25 12:22:21 +02:00
andri lim 70be982259
Fix CI badge URL (#2150) 2024-04-24 08:17:13 +07:00
andri lim 47605dbdf3
Fix txPool: excessBlobGas should be calculated before entering EVM, not after (#2149) 2024-04-24 07:50:50 +07:00
Jordan Hrycaj b9187e0493
Aristo selective read cashing for rocksdb backend (#2145)
* Aristo+Kvt: Better RocksDB profiling

why:
  Providing more detailed information, mainly for `Aristo`

* Aristo: Renamed journal `stats()` to `capacity()`

why:
  `Stats()` was a misnomer

* Aristo: Provide backend read caches for key and vertex IDs

why:
  Dedicated LRU caching for particular types gives a throughput advantage.
  The sizes of the LRU queues used for caching are currently constant
  but might be adjusted at a later time.

* Fix copyright year
2024-04-22 19:02:22 +00:00
Kim De Mey 1d90b924ee
Adjust color scheme for fluffy.guide (#2144)
Use the same scheme as for the nimbus.guide.
Two differences however:
- Set the dark scheme as default, considering the new nimbus.team
website also has dark as default.
- Remove the orange/amber for the light scheme, considering the
light version of nimbus.team has no more orange.
2024-04-22 15:33:51 +02:00
andri lim 4078cb14c5
Engine api simulator: fix payload customizer (#2143)
* Engine api simulator: fix payload customizer

* Move versionedHashes validation before blockHash validation in newPayload

* More descriptive error message

* Fix Web3Hash to eth.Hash256 conversion
2024-04-21 21:44:05 +07:00
andri lim 6694e240d7
Repositioning blob hash validation in newPayload of engine API (#2141) 2024-04-20 02:43:13 +07:00
Jordan Hrycaj 7d9e1d8607
Misc updates for full sync (#2140)
* Code cosmetics

* Aristo+Kvt: Fix api wrappers

why:
  Api setup killed the backend descriptor when backend mapping was
  disabled.

* Aristo: Implement masked profiling entries

why:
  Database backend should be listed but not counted in tally

* CoreDb: Simplify backend() methods

why:
  DBMS backend access Was provided very early and over engineered. Now
  there are only two backend machines, one for `Kvt` and the other one
  for an `Mpt` available only via new API.

* CoreDb: Code cleanup regarding descriptor types

* CoreDb: Refactor/redefine `persistent()` methods

why:
  There were `persistent()` methods for any type of caching storage
  facilities `Kvt`, `Mpt`, `Phk`, and `Acc`. Now there is only a single
  `persistent()` method storing all facilities in tandem (similar to
  how transactions work.)

  For non shared `Kvt` tables, there is now an extra storage method
  `saveOffSite()`.

* CoreDb lingo update: `trie` becomes `column`

why:
  Notion of a `trie` is pretty much hidden by the new `CoreDb` api.
  Revealed are sort of database columns for accounts an storage data,
  any of which have an internal state represented by a Keccack hash.
  So a `trie` or `MPT` becomes a `column` and a `rootHash` becomes a
  column state.

* Aristo: rename backend filed `filters` => `journal`

* Update full sync logging

details:
  + Disable eth handler noise while syncing
  + Log journal depth (if available)

* Fix copyright year

* Fix cruft and unwanted imports
2024-04-19 18:37:27 +00:00
jangko e2c873b16f
Fix engine API and engine API simulator according to latest spec. 2024-04-18 14:55:49 +07:00
Jordan Hrycaj e8eb3268f5
Generalise prune mode option 4 different db models (#2139)
* Update README

* Nimbus-main: replaced `PruneMode` options by `ChainDbMode` options

details:
  For the legacy database, this changes the phrase
  - `conf.pruneMode == PruneMode.Full` to the expression
  + `conf.chainDbMode == ChainDbMode.Prune`.

* Fix issues moaned about by NIM compiler

* Fix copyright year
2024-04-17 18:09:55 +00:00
Kim De Mey c610053a09
Prepare Fluffy for usage of historical_roots (#2138) 2024-04-17 17:58:47 +02:00