Commit Graph

207 Commits

Author SHA1 Message Date
Kim De Mey e603952df3
Refactor portal test directory structure (#2650) 2024-09-24 13:07:20 +02:00
Kim De Mey 0fb9581b96
Refactor and apply renaming to post merge block proofs (#2648) 2024-09-23 18:56:28 +02:00
web3-developer 07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
web3-developer 0ee8e61a3a
Remove RpcProxy from Fluffy and use RpcHttpServer instead. (#2634)
* Remove RpcProxy from Fluffy and use RpcHttpServer instead.

* Cleanup test.
2024-09-18 15:46:50 +08:00
web3-developer b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer 9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey 45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
Kim De Mey 0869a27462
Move inRange check to the Portal protocol + related simplifications (#2602) 2024-09-09 17:52:11 +02:00
Kim De Mey cb94dd0c5b
Small cleanup of the ContentDB store handler (#2597) 2024-09-07 15:03:13 +02:00
Kim De Mey e919f57902
Remove old header network code as this no longer exists in specs (#2596) 2024-09-06 18:00:58 +02:00
Kim De Mey 0a80a3bb25
Reverse calculate the radius at node restart (#2593)
This avoid restarting the node always with a full radius, which
causes the node the be bombarded with offers which it later has
to delete anyhow.

In order to implement this functionality, several changes were
made as the radius needed to move from the Portal wire protocol
current location to the contentDB and beaconDB, which is
conceptually more correct anyhow.

So radius is now part of the database objects and a handler is
used in the portal wire protocol to access its value.
2024-09-05 18:31:55 +02:00
web3-developer fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer 507a9e71df
Fluffy state network fixes and improvements (#2576)
* Cleanup tests.

* Improve offer sort function in state bridge.

* Verify nibble prefix for leaf and extension nodes during lookup.
2024-08-23 15:46:23 +08:00
web3-developer c8d34eba9b
Bump portal-spec-tests and update Fluffy state tests (#2568)
* Bump portal-spec-tests.

* Update state network tests to use block_header instead of state_root.
2024-08-19 14:45:54 +08:00
web3-developer 93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
web3-developer 947f629903
Fluffy State Bridge - State Gossip via Portal JSON-RPC (#2535)
* Create block offers queue and collect account preimages.

* Implement iterators to return account and storage proofs and bytecode from updatedCaches.

* Implement building offers from proofs.

* Refactor BlockDataRef type to only include required fields.

* Store block data in database.

* Improve state diff types.

* Implement start state backfill from specific block.

* Record last persisted block number in database.

* Persist account preimages in db.

* Apply state updates for DAO hard fork.

* Implement state gossip of block offers via portal JSON RPC.
2024-07-30 22:56:21 +08:00
Kim De Mey 7e2a636717
Add basic validation for LC bootstraps + portal_bridge changes (#2527)
- Add basic validation for LC bootstrap gossip, validating either
by trusted block root (only 1) when not synced, or by comparing
with the header of the latest finality update when synced.

- Update portal_bridge beacon to also gossip bootstraps into the
network on each end of epoch.
2024-07-25 20:15:26 +02:00
Kim De Mey 84ce0c912e
Bump Portal test vectors and re-activate skipped tests (#2515) 2024-07-23 11:59:59 +02:00
Kim De Mey 51cf991439
Bump ssz_serialization and use ByteList[n] + add ContentKeyByteList (#2500) 2024-07-17 17:07:27 +02:00
Kim De Mey 3bb707422b
Rework beacon block proofs to better structure (#2493)
The 3 proofs can be reworked to two proofs as we can use the
BeaconBlock directly instead of BeaconBlockHeader and
BeaconBlockBody. This is possible because the HTR of the
BeaconBlock is the same as the one of the BeaconBlockHeader.

This results in 32 bytes less as an intermediate hash can be
removed. But more importantly looks more clean and compact in
structure and code.
2024-07-17 11:32:05 +02:00
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey 54e3fd1a94
Move Portal wire and networks setup to new portal_node module (#2464) 2024-07-09 19:22:25 +02:00
Kim De Mey ae094692fb
Bump nim-eth for ENR changes and related clean-up (#2422) 2024-06-27 17:59:08 +02:00
web3-developer 09946c9958
Fluffy state network tests and logging improvements (#2402)
* Improve tests by waiting for content to get into db.

* Improve state network logging.
2024-06-20 23:41:34 +08:00
web3-developer 0b8363764d
More state endpoint tests (#2399) 2024-06-20 00:21:23 +08:00
Kim De Mey 4fd2ecddec
Bump nim-eth/web3/kzg4844/nimbus-eth2 and related fixes (#2392)
Bump nim-eth, which requires nimbus-eth2 bump, which requires
bumps of web3 and kzg4844 + related fixes to all those bumps.
2024-06-19 08:57:45 +07:00
Miran ea0d18424a
use Nim 2.0.6 (#2384)
* use Nim 2.0.6

* Fixes for nim 2.0.6

* Workaround nim 2.0 array indexing issue

* Remove excess gcsafe pragma

* Oops, fix recursive template

* Fix imports

* Fluffy nph linting

---------

Co-authored-by: jangko <jangko128@gmail.com>
Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-19 01:27:54 +00:00
web3-developer e3d14bd921
Fluffy portal testnet support (#2383)
* Bump portal-mainnet repo.

* Update command line arguments and parsing on startup.

* Read in angelfood bootstrap nodes and update Fluffy guide.

* Configure subnetwork protocol ids.
2024-06-18 15:32:57 +08:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
web3-developer f326ae2ee1
Cleanup state network recursive gossip which was removed from the portal specs. (#2345) 2024-06-13 13:53:49 +08:00
web3-developer 9c26fa3298
Updates to Fluffy to support hive tests. (#2333)
* Updates to Fluffy book for hive tests.

* Add support to disable state root checks for state content from the command line.

* Update portal_stateStore endpoint to support decoding offer and storing retrieval value.
2024-06-11 21:01:35 +08:00
web3-developer c72d6aa5d6
Implementation of procs to support Fluffy state JSON-RPC endpoints. (#2318)
* Started implementation of state endpoints.

* Add rpc calls and server stubs.

* Initial implementation of getAccountProof and getStorageProof.

* Refactor validation to use toAccount utils functions.

* Add state endpoints tests.
2024-06-10 18:47:09 +08:00
Jacek Sieka f6be4bd0ec
avoid initTable (#2328)
`initTable` is obsolete since nim 0.19 and can introduce significant
memory overhead while providing no benefit (since the table will be
grown to the default initial size on first use anyway).

In particular, aristo layers will not necessarily use all tables they
initialize, for exampe when many empty accounts are being created.
2024-06-10 11:05:30 +02:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
web3-developer 1eb170e686
Fluffy state network tests (#2245)
* Add state network get content test.

* Completed state network get content tests.

* Completed state gossip offer content test.

* Improve state network offer content test.

* Completed state gossip offer test.

* Rename tests.
2024-05-30 17:00:57 +08:00
web3-developer 9354cb8411
Fluffy recursive gossip improvements, fixes and tests. (#2231)
* Refactor get parent gossip code and add Nibbles helper function.

* Add logging to state gossip.

* Unit test recursive gossip using state gossip getParent functions.

* Add recursive gossip genesis json test and fix bug in state gossip getParent.

* Add Nibbles len function.
2024-05-28 08:45:30 +08:00
Kim De Mey 72912626a2
Several spec fixes for HistoricalSummaries + add test-vector test (#2217) 2024-05-24 23:15:04 +02:00
web3-developer 771d6fd3e8
Implementation of Fluffy state network gossip. (#2210)
* Add toContentKey helper functions.

* Implement typed lookup functions for getting account and storage trie nodes and bytecode.

* Re-implementation of recursive gossip.

* Offer validation improvements. Don't allow key end in Nibbles for leafs as defined in the spec.

* Minor refactor to validation and add addition bytecode check.
2024-05-24 15:49:51 +08:00
web3-developer e566e89d03
Fluffy state network refactor (#2200)
* Add additional getTrieProof test.

* Refactor state network: Removed variant objects for values and split state_content into multiple files.
2024-05-22 09:22:07 +08:00
web3-developer 047f8ec4b2
Bump portal spec tests and update state validation tests to use state roots in test vectors. (#2193)
* Bump portal spec tests to version a549242

* Update state validation tests to use state roots in portal spec test vectors.
2024-05-17 23:20:57 +08:00
Kim De Mey f1d7d2d62b
Support multiple test vectors to be run for HeaderWithProof tests (#2191) 2024-05-16 13:37:56 +02:00
web3-developer 2891b9aa7d
Fluffy state offer validation (#2170)
* Implemented offer validation interface.

* Implement high level offer validation steps.

* Completed state validation tests.

* Update validation to use result type.

* Update state proof verification tests to test offer verification.

* Query history network to get state root by block hash.

* Fix state network test and remove usage of CoreDb.

* Fix state network gossip test and PR comment updates.

* Add trieproof state validation tests and fix for short nodes.
2024-05-16 16:16:16 +08:00
Kim De Mey 2e01878afe
Add the epoch to the HistoricalSummariesKey (#2179) 2024-05-13 18:49:21 +02:00
Kim De Mey 3be298f58f
Add tests for beacon block proofs Capella and onwards (#2161) 2024-05-10 22:51:24 +02:00
web3-developer 5c4358a969
Add retrieval content validation to state network. (#2166)
* Add retrieval content validation to state network.

* Complete state validation tests for retrieval content.

* Improve naming functions and parameters.
2024-05-06 23:13:35 +08:00
web3-developer 7da36bf459
Minor refactor to packNibbles and unpackNibbles and add additional tests. (#2162) 2024-05-03 22:55:51 +08:00
Kim De Mey df2dcf593c
Remove few more unneeded nimbus imports in Fluffy tests (#2159) 2024-05-03 12:31:54 +02:00
Kim De Mey 9c64b44c89
Add post Bellatrix beacon block proof exports (#2158)
- Add post Bellatrix Beacon block proof exporting to
eth_data_exporter
- Rework the Bellatrix Beacon block proof exports to us yaml
format
- Move some common code of the two different beacon block proofs
to beacon_chain_block_proof_common module
- Move + add to yaml utils to be usable not just for the tests but
also for eth data exporter
2024-05-03 11:50:18 +02:00
web3-developer d12300ecc9
Remove usages of CoreDb in Fluffy. (#2152)
* Remove references to CoreDb in Fluffy code.

* Add missing import.

* Remove coredb imports and use dynamic server port in tests.

* Fix formatting using nph.
2024-04-25 12:22:21 +02:00
Kim De Mey c610053a09
Prepare Fluffy for usage of historical_roots (#2138) 2024-04-17 17:58:47 +02:00