Commit Graph

133 Commits

Author SHA1 Message Date
bhartnett 54528fb24b
Fluffy: Enable content cache for state network (#2739)
* Enable content cache for state network.

* Update state json-rpc endpoints to return local content which uses cache if enabled.

* Add content cache metrics.

* Make content cache configurable.

* Add content cache tests.
2024-10-16 21:05:39 +08:00
andri lim 1126c7700d
Bump nim-eth and nimbus-eth2 (#2741)
* Bump nim-eth and nimbus-eth2

* Fix ambiguous identifier
2024-10-16 13:51:38 +07:00
bhartnett bdbea9853e
Rename Fluffy rpc methods to match updated spec (#2735) 2024-10-14 16:46:35 +08:00
Kim De Mey 6e114dc950
Fix JSON encoding for NodeId, no leading zeroes dropped (#2730)
Portal JSON-RPC API specifications dictate that the NodeId must be
the 32 byte identifier. We had leading zeroes being dropped in
our implementation.
2024-10-11 16:43:06 +02:00
Kim De Mey a5f0b12bd1
Remove old portal_debug API for seeding data into the network (#2726)
This was the first API created for this, it has been superseded
by the API that makes use of era1 files and/or the portal_bridge.

Only usage was still in test_portal_testnet which has now been
altered to make use if API calls from Portal specification.
2024-10-10 16:42:57 +02:00
bhartnett 4ae87e6d19
Fluffy: Add validation and local storage of content in remaining state portal rpc methods (#2723)
* Add validation functions to be used in state portal rpc.

* Add validation to remaining state portal rpc methods.

* Lookup local content in recursiveFindContent rpc methods.

* portal_stateFindContent and portal_stateOffer no longer store in db.
2024-10-10 21:24:39 +08:00
bhartnett b13f06fcfb
Fluffy state portal rpc validation (#2719)
* Validate content key in portal_stateLocalContent.

* Add additional validation to stateStore rpc method.
2024-10-09 20:23:46 +08:00
bhartnett 72ee610826
Rename Fluffy debug rpc methods (#2710)
* Rename Fluffy debug rpc methods.

* Organize portal json-rpc errors.
2024-10-08 15:21:27 +08:00
Kim De Mey a3bb5d4428
Small clean-up of the differen Portal json-rpc APIs (#2708)
Clean-up removing:
- no longer relevant comments
- unused types
- unused imports
2024-10-08 09:15:29 +08:00
Kim De Mey 833719a866
Remove usage of aliases for Hash32 such as BlockHash (#2707)
These create only confusion as if they are actual different types
and it is within their usage already clear what they are about
because of the name of the variable or the function.

They are also nowhere aliased like this in any of the Portal
specification.
2024-10-07 22:39:07 +02:00
Kim De Mey e1c942ccef
Remove unneeded w3Hash and similar calls (#2706) 2024-10-07 20:54:48 +02:00
bhartnett 344a236c31
Fluffy: Create separate portal rpc handlers for each sub-network (#2705)
* Create separate portal rpc handlers for each sub-network.

* Remove random gossip from history and state networks.

* Move shared portal rpc endpoints into common handler.

* Use enum instead of network string.
2024-10-07 16:33:02 +02:00
Kim De Mey 778c1d589f
Adapt portal_*Offer method to the latest specification (#2689) 2024-10-07 10:49:04 +02:00
Kim De Mey dbe3393f5c
Fix eth/common & web3 related deprecation warnings for fluffy (#2698)
* Fix eth/common & web3 related deprecation warnings for fluffy

This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.

It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.

There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.

* Address review comments

* Remove two more unused eth/common imports
2024-10-04 23:21:26 +02:00
Jacek Sieka 08ffb3161c
Use eth/common transaction signature utilities (#2696)
* Use eth/common transaction signature utilities

* bump

* bump

* bump

* bump

* bump

* bump
2024-10-04 16:34:31 +02:00
Kim De Mey 0b81a8c783
Adjust error code on portal_*TraceRecursiveFindContent (#2687) 2024-10-03 14:25:49 +02:00
bhartnett 9192aa13ed
Fluffy state bridge updates (#2683)
* Use Json decode when handling portal rpc client error.

* Remove preimages cache.

* Support disabling state gossip in order to just build state.
2024-10-03 10:57:19 +08:00
bhartnett 44fea2348b
Fluffy Portal RPC Client (#2649)
* Portal rpc client implementation.
2024-10-02 12:23:13 +08:00
Kim De Mey 5e3f3db165
Remove client-side json-rpc calls that were removed server-side (#2659) 2024-09-25 22:35:19 +02:00
web3-developer 69d58e8215
Fluffy: Configure RPC APIs via CLI parameter (#2657)
* Support RPC API namespaces as cli parameter.

* Fluffy now uses rpcFlags on startup.

* Update testnet script to enable all RPC APIs.

* Update Fluffy book and move web3 call into eth calls.
2024-09-25 22:44:46 +08:00
web3-developer b1dc1578f0
Move web3 rpc handler into eth handler. Create debug rpc handler and move existing debug endpoints. (#2655) 2024-09-25 11:28:45 +08:00
web3-developer 0ee8e61a3a
Remove RpcProxy from Fluffy and use RpcHttpServer instead. (#2634)
* Remove RpcProxy from Fluffy and use RpcHttpServer instead.

* Cleanup test.
2024-09-18 15:46:50 +08:00
web3-developer c652f5efc2
Fluffy state bridge docs (#2632)
* Don't proxy implemented state JSON-RPC endpoints. Fix minor issue in testnet script.

* Add docs covering usage of the Fluffy state bridge.
2024-09-18 11:56:20 +08:00
web3-developer b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer 9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey 45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
web3-developer ee6a7e8259
Fluffy state endpoint improvements (#2580)
* Return default values when account, slot or code doesn't exist.

* Handle case when storage doesn't exist due to account not existing or being a non contract account.
2024-08-28 16:27:36 +08:00
web3-developer fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer f9956eba59
Fluffy State Bridge - Building state using state diffs. (#2486)
* Started state bridge.

* Implement call to fetch stateDiffs using trace_replayBlockTransactions.

* Convert JSON responses to stateDiff types.

* State updates working for first few blocks.

* Correctly building state for first 200K blocks.

* Add storage of code and cleanup.

* Start state bridge refactor.

* More cleanup and fixes.

* Use RocksDb as backend for state.

* Implement transactions.

* Build RocksDb dependency when building fluffy tools.

* Move code to world state helper.

* Implement producer and consumer queue.

* Cleanup exceptions.

* Improve logging.

* Add update caches to DatabaseRef backends.
2024-07-18 17:01:40 +08:00
Kim De Mey 51cf991439
Bump ssz_serialization and use ByteList[n] + add ContentKeyByteList (#2500) 2024-07-17 17:07:27 +02:00
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey 4a20756e6b
Remove unused seed_db and related code (#2471) 2024-07-10 23:02:15 +02:00
Kim De Mey ae094692fb
Bump nim-eth for ENR changes and related clean-up (#2422) 2024-06-27 17:59:08 +02:00
web3-developer e3d14bd921
Fluffy portal testnet support (#2383)
* Bump portal-mainnet repo.

* Update command line arguments and parsing on startup.

* Read in angelfood bootstrap nodes and update Fluffy guide.

* Configure subnetwork protocol ids.
2024-06-18 15:32:57 +08:00
web3-developer 1377f93d50
Implement Fluffy JSON-RPC endpoints. (#2364)
* Implement eth_getBalance, eth_getTransactionCount, eth_getStorageAt and eth_getCode JSON-RPCs.

* Fixes.
2024-06-15 01:03:18 +08:00
Jacek Sieka 8a0772ac10
cleanup a few more rlpHash calls (#2359) 2024-06-14 17:42:31 +07:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
web3-developer 9c26fa3298
Updates to Fluffy to support hive tests. (#2333)
* Updates to Fluffy book for hive tests.

* Add support to disable state root checks for state content from the command line.

* Update portal_stateStore endpoint to support decoding offer and storing retrieval value.
2024-06-11 21:01:35 +08:00
web3-developer c72d6aa5d6
Implementation of procs to support Fluffy state JSON-RPC endpoints. (#2318)
* Started implementation of state endpoints.

* Add rpc calls and server stubs.

* Initial implementation of getAccountProof and getStorageProof.

* Refactor validation to use toAccount utils functions.

* Add state endpoints tests.
2024-06-10 18:47:09 +08:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
Kim De Mey 1512f95067
Remove unneeded nimbus imports (#2154)
Also specifically select from the still required
nimbus imports.
2024-04-25 15:21:09 +02:00
Kim De Mey 8e4368195a
Add an audit mode to portal_bridge history backfill (#2109) 2024-03-26 22:27:31 +01:00
andri lim 30277be1f3
Bump nim-web3 to 285d97c2b05bbe2a13dab4b52ea878157fb1a1a1 (#2088)
* Bump nim-web3 to 285d97c2b05bbe2a13dab4b52ea878157fb1a1a1

Unify EthCall/EthSend into TransactionArgs (#138)

* bump ssz-serialization

* Fix BlockNumber conversion

* Bump ssz-serialization: Restrict toSszType usage to non SszType in readSszBytes (#81)
2024-03-21 08:05:22 +07:00
Kim De Mey bfa756c36b
Bump json-rpc and web3 and remove unneeded errors import/export (#2078) 2024-03-15 11:44:18 +01:00
Kim De Mey 461c7e7d85
Add Era1 backfill to portal_bridge history mode (#2077) 2024-03-15 10:41:41 +01:00
Kim De Mey 7287efc7b4
Clean-up of some Portal json-rpc code (#2073)
- Use the new createRpcSigsFromNim for client json-rpc API
- Avoid importing any nimbus/rpc specifics, use only web3 and
fluffy local rpc code
- Adjust tools making use of the client side API
2024-03-13 16:58:50 +01:00
Kim De Mey 88a93beb26
Provide a specific error in JSON-RPC API when content is not found (#2059) 2024-03-06 08:57:09 +01:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
Kim De Mey 9378774b9f
Add Era1 based gossip calls + Era1 helpers (#2029)
- Add Era1 helpers to be able to iterate fast over block tuples
and individual total difficulties
- Add buildAccumulator from Era1 file
- Add Era1 based BlockHeader with proof + bodies/receipts gossip
calls
- Add new JSON-RPC debug methods to be able to test the above
with a standalone fluffy node
2024-02-15 16:49:22 +01:00
Kim De Mey 3199857ed5
Further fluffy code clean-up from warnings/hints (#1987) 2024-01-25 11:04:09 +01:00