Commit Graph

287 Commits

Author SHA1 Message Date
web3-developer 07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
Kim De Mey c77c6239b0
Use discv5's maxDiscv5TalkRespPayload const for payload size calc (#2643) 2024-09-19 20:30:10 +02:00
web3-developer a9ad10cadc
Fluffy state network now enabled by default and improve status logs (#2640)
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
2024-09-19 21:38:49 +08:00
web3-developer b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer 9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey 45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
Kim De Mey 4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
Kim De Mey 0869a27462
Move inRange check to the Portal protocol + related simplifications (#2602) 2024-09-09 17:52:11 +02:00
Kim De Mey e919f57902
Remove old header network code as this no longer exists in specs (#2596) 2024-09-06 18:00:58 +02:00
Kim De Mey 0a80a3bb25
Reverse calculate the radius at node restart (#2593)
This avoid restarting the node always with a full radius, which
causes the node the be bombarded with offers which it later has
to delete anyhow.

In order to implement this functionality, several changes were
made as the radius needed to move from the Portal wire protocol
current location to the contentDB and beaconDB, which is
conceptually more correct anyhow.

So radius is now part of the database objects and a handler is
used in the portal wire protocol to access its value.
2024-09-05 18:31:55 +02:00
web3-developer ee6a7e8259
Fluffy state endpoint improvements (#2580)
* Return default values when account, slot or code doesn't exist.

* Handle case when storage doesn't exist due to account not existing or being a non contract account.
2024-08-28 16:27:36 +08:00
web3-developer fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer 507a9e71df
Fluffy state network fixes and improvements (#2576)
* Cleanup tests.

* Improve offer sort function in state bridge.

* Verify nibble prefix for leaf and extension nodes during lookup.
2024-08-23 15:46:23 +08:00
web3-developer 93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
web3-developer 63d13182c1
Fluffy state bridge - Add CLI parameters and implement workers for offer gossip. (#2541)
* Use RPC batching to send offer requests and filter out duplicate offers.

* Lookup offers after gossip to check if gossip successful.

* Use multiple workers for gossiping offers.

* Update Fluffy state network logging.

* Use single RPC calls instead of batching.

* Update cli parameters.

* Fix bug in contract trie offer building.
2024-08-06 15:38:38 +08:00
web3-developer 947f629903
Fluffy State Bridge - State Gossip via Portal JSON-RPC (#2535)
* Create block offers queue and collect account preimages.

* Implement iterators to return account and storage proofs and bytecode from updatedCaches.

* Implement building offers from proofs.

* Refactor BlockDataRef type to only include required fields.

* Store block data in database.

* Improve state diff types.

* Implement start state backfill from specific block.

* Record last persisted block number in database.

* Persist account preimages in db.

* Apply state updates for DAO hard fork.

* Implement state gossip of block offers via portal JSON RPC.
2024-07-30 22:56:21 +08:00
Kim De Mey d4d8d2af64
Add validation on gossip of LC updates as per spec (#2528) 2024-07-26 18:56:32 +02:00
Kim De Mey 7e2a636717
Add basic validation for LC bootstraps + portal_bridge changes (#2527)
- Add basic validation for LC bootstrap gossip, validating either
by trusted block root (only 1) when not synced, or by comparing
with the header of the latest finality update when synced.

- Update portal_bridge beacon to also gossip bootstraps into the
network on each end of epoch.
2024-07-25 20:15:26 +02:00
Kim De Mey 3a7f025fd9
Small cleanup/refactor on Portal history network (#2521) 2024-07-24 20:42:12 +02:00
Kim De Mey 1841fe7c4a
Remove old workaround for toString that was import-leaking in (#2520) 2024-07-24 15:33:43 +02:00
Kim De Mey 45d85aa3b5
Add HistoricalSummariesWithProof gossip to portal_bridge beacon (#2514) 2024-07-22 19:07:46 +02:00
Kim De Mey ff40c1e1a2
Update nph to 0.6.0 for fluffy and nimbus_verified_proxy (#2511) 2024-07-22 14:22:45 +02:00
Kim De Mey 51cf991439
Bump ssz_serialization and use ByteList[n] + add ContentKeyByteList (#2500) 2024-07-17 17:07:27 +02:00
Kim De Mey 3bb707422b
Rework beacon block proofs to better structure (#2493)
The 3 proofs can be reworked to two proofs as we can use the
BeaconBlock directly instead of BeaconBlockHeader and
BeaconBlockBody. This is possible because the HTR of the
BeaconBlock is the same as the one of the BeaconBlockHeader.

This results in 32 bytes less as an intermediate hash can be
removed. But more importantly looks more clean and compact in
structure and code.
2024-07-17 11:32:05 +02:00
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey 4a20756e6b
Remove unused seed_db and related code (#2471) 2024-07-10 23:02:15 +02:00
Kim De Mey 94340037bf
Set the routing table ip limits back to defaults (#2470) 2024-07-10 17:26:30 +02:00
Kim De Mey 54e3fd1a94
Move Portal wire and networks setup to new portal_node module (#2464) 2024-07-09 19:22:25 +02:00
andri lim d9e502bbc5
Bump web3/kzg4844/nimbus-eth2 and related fixes (#2446) 2024-07-04 05:41:32 +00:00
Kim De Mey 87d090afa6
Refactor history/beacon content types to seperate keys and values (#2438)
Refactor content types, to have similar structure as for state
network.
Amd some clean-up of comments/TODOs/spec links.
2024-07-02 15:48:49 +02:00
Kim De Mey ae094692fb
Bump nim-eth for ENR changes and related clean-up (#2422) 2024-06-27 17:59:08 +02:00
web3-developer 09946c9958
Fluffy state network tests and logging improvements (#2402)
* Improve tests by waiting for content to get into db.

* Improve state network logging.
2024-06-20 23:41:34 +08:00
web3-developer bd2ca07da6
Update Fluffy data directory location based on selected network. (#2400) 2024-06-20 10:48:45 +08:00
web3-developer 0b8363764d
More state endpoint tests (#2399) 2024-06-20 00:21:23 +08:00
Miran ea0d18424a
use Nim 2.0.6 (#2384)
* use Nim 2.0.6

* Fixes for nim 2.0.6

* Workaround nim 2.0 array indexing issue

* Remove excess gcsafe pragma

* Oops, fix recursive template

* Fix imports

* Fluffy nph linting

---------

Co-authored-by: jangko <jangko128@gmail.com>
Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-19 01:27:54 +00:00
web3-developer e3d14bd921
Fluffy portal testnet support (#2383)
* Bump portal-mainnet repo.

* Update command line arguments and parsing on startup.

* Read in angelfood bootstrap nodes and update Fluffy guide.

* Configure subnetwork protocol ids.
2024-06-18 15:32:57 +08:00
web3-developer 1377f93d50
Implement Fluffy JSON-RPC endpoints. (#2364)
* Implement eth_getBalance, eth_getTransactionCount, eth_getStorageAt and eth_getCode JSON-RPCs.

* Fixes.
2024-06-15 01:03:18 +08:00
Kim De Mey c31fc37c62
Add async raises annotations for Portal wire and Portal networks (#2361) 2024-06-14 14:21:30 +02:00
Jacek Sieka 8a0772ac10
cleanup a few more rlpHash calls (#2359) 2024-06-14 17:42:31 +07:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
web3-developer 7fd777cfa9
Add push raises to Fluffy state network code. (#2352)
* Add push raises to Fluffy state network code.

* Refactor handling of rlp errors.
2024-06-14 13:38:24 +08:00
web3-developer f326ae2ee1
Cleanup state network recursive gossip which was removed from the portal specs. (#2345) 2024-06-13 13:53:49 +08:00
web3-developer 9c26fa3298
Updates to Fluffy to support hive tests. (#2333)
* Updates to Fluffy book for hive tests.

* Add support to disable state root checks for state content from the command line.

* Update portal_stateStore endpoint to support decoding offer and storing retrieval value.
2024-06-11 21:01:35 +08:00
web3-developer c72d6aa5d6
Implementation of procs to support Fluffy state JSON-RPC endpoints. (#2318)
* Started implementation of state endpoints.

* Add rpc calls and server stubs.

* Initial implementation of getAccountProof and getStorageProof.

* Refactor validation to use toAccount utils functions.

* Add state endpoints tests.
2024-06-10 18:47:09 +08:00
Jacek Sieka f6be4bd0ec
avoid initTable (#2328)
`initTable` is obsolete since nim 0.19 and can introduce significant
memory overhead while providing no benefit (since the table will be
grown to the default initial size on first use anyway).

In particular, aristo layers will not necessarily use all tables they
initialize, for exampe when many empty accounts are being created.
2024-06-10 11:05:30 +02:00
Kim De Mey d814d84b9b
Use --styleCheck:error for Fluffy + fixes (#2247)
* Use --styleCheck:error for Fluffy + fixes

There seems to be a clash between the names of an object field and
a proc here. As workaround names of the procs are changed.

* Fix case style for rpc client calls in test_portal_testnet

* Fix style case for utp test
2024-05-30 18:12:28 +02:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
web3-developer 1eb170e686
Fluffy state network tests (#2245)
* Add state network get content test.

* Completed state network get content tests.

* Completed state gossip offer content test.

* Improve state network offer content test.

* Completed state gossip offer test.

* Rename tests.
2024-05-30 17:00:57 +08:00
web3-developer 9354cb8411
Fluffy recursive gossip improvements, fixes and tests. (#2231)
* Refactor get parent gossip code and add Nibbles helper function.

* Add logging to state gossip.

* Unit test recursive gossip using state gossip getParent functions.

* Add recursive gossip genesis json test and fix bug in state gossip getParent.

* Add Nibbles len function.
2024-05-28 08:45:30 +08:00
Kim De Mey 72912626a2
Several spec fixes for HistoricalSummaries + add test-vector test (#2217) 2024-05-24 23:15:04 +02:00