Commit Graph

481 Commits

Author SHA1 Message Date
web3-developer ea74e035ab
Add WebSocket support to Fluffy. (#2636)
* Add WebSocket support to Fluffy.

* Support websocket compression.

* Create setupRpcServer closure to remove code duplication.
2024-09-18 21:27:24 +08:00
web3-developer 0ee8e61a3a
Remove RpcProxy from Fluffy and use RpcHttpServer instead. (#2634)
* Remove RpcProxy from Fluffy and use RpcHttpServer instead.

* Cleanup test.
2024-09-18 15:46:50 +08:00
web3-developer c652f5efc2
Fluffy state bridge docs (#2632)
* Don't proxy implemented state JSON-RPC endpoints. Fix minor issue in testnet script.

* Add docs covering usage of the Fluffy state bridge.
2024-09-18 11:56:20 +08:00
web3-developer d17ddacf39
More improvements to the Fluffy local testnet script (#2624)
* More improvements to the Fluffy local testnet script.
2024-09-13 20:40:48 +08:00
Kim De Mey d96196d01d
Add support for BlockHeader by number in portal_bridge (#2621) 2024-09-13 13:41:34 +02:00
web3-developer b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer 9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey 45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
Kim De Mey fd3475ea3e
Add a brief doc page on how to add documentation (#2617) 2024-09-12 10:50:45 +02:00
Kim De Mey 4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
web3-developer 3a73b948c6
Fluffy testnet script updates (#2615)
* Update local testnet script to support additional options.

* Increase state bridge retry sleep times.

* Fix run fluffy testnet tests in CI.
2024-09-12 16:18:24 +08:00
tersec 1b173d420d
small cleanups (#2598)
* small cleanups

* stop hiding ConvFromXtoItselfNotNeeded hints

* lowmem optimization flag is no-op
2024-09-10 05:24:45 +00:00
Kim De Mey 0869a27462
Move inRange check to the Portal protocol + related simplifications (#2602) 2024-09-09 17:52:11 +02:00
Kim De Mey cb94dd0c5b
Small cleanup of the ContentDB store handler (#2597) 2024-09-07 15:03:13 +02:00
Kim De Mey e919f57902
Remove old header network code as this no longer exists in specs (#2596) 2024-09-06 18:00:58 +02:00
Kim De Mey 8a19118ddf
Fix initial radius bug due to uninitialized localId (#2595) 2024-09-06 11:45:27 +02:00
Kim De Mey 0a80a3bb25
Reverse calculate the radius at node restart (#2593)
This avoid restarting the node always with a full radius, which
causes the node the be bombarded with offers which it later has
to delete anyhow.

In order to implement this functionality, several changes were
made as the radius needed to move from the Portal wire protocol
current location to the contentDB and beaconDB, which is
conceptually more correct anyhow.

So radius is now part of the database objects and a handler is
used in the portal wire protocol to access its value.
2024-09-05 18:31:55 +02:00
web3-developer 139b35a80a
Fluffy State Bridge: State building fixes (#2589)
* Fix bug where code was not correctly being updated during contract creation.

* Delete value from update cache during db delete.
2024-09-03 21:05:38 +08:00
web3-developer ee6a7e8259
Fluffy state endpoint improvements (#2580)
* Return default values when account, slot or code doesn't exist.

* Handle case when storage doesn't exist due to account not existing or being a non contract account.
2024-08-28 16:27:36 +08:00
web3-developer fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer 507a9e71df
Fluffy state network fixes and improvements (#2576)
* Cleanup tests.

* Improve offer sort function in state bridge.

* Verify nibble prefix for leaf and extension nodes during lookup.
2024-08-23 15:46:23 +08:00
web3-developer 60c9b2c00d
Fluffy State Bridge: Fetch portal client nodeId on startup and sort offers by distance from nodeId (#2570)
* Fetch portal client nodeId on startup and sort offers by distance from nodeId.

* Fix logging.

* Improve error handling at startup.
2024-08-19 20:49:07 +08:00
web3-developer c8d34eba9b
Bump portal-spec-tests and update Fluffy state tests (#2568)
* Bump portal-spec-tests.

* Update state network tests to use block_header instead of state_root.
2024-08-19 14:45:54 +08:00
Jordan Hrycaj cbe5131927
Simplify aristo tree deletion functionality (#2563)
* Cleaning up, removing cruft and debugging statements

* Make `aristo_delta` fluffy compatible

why:
  A sub-module that uses `chronicles` must import all possible
  modules used by a parent module that imports the sub-module.

* update TODO
2024-08-14 12:09:30 +00:00
Jordan Hrycaj ce713d95fc
Aristo lazily delete larger subtrees (#2560)
* Extract sub-tree deletion functions into separate sub-modules

* Move/rename `aristo_desc.accLruSize` => `aristo_constants.ACC_LRU_SIZE`

* Lazily delete sub-trees

why:
  This gives some control of the memory used to keep the deleted vertices
  in the cached layers. For larger sub-trees, keys and vertices might be
  on the persistent backend to a large extend. This would pull an amount
  of extra information from the backend into the cached layer.

  For lazy deleting it is enough to remember sub-trees by a small set of
  (at most 16) sub-roots to be processed when storing persistent data.
  Marking the tree root deleted immediately allows to let most of the code
  base work as before.

* Comments and cosmetics

* No need to import all for `Aristo` here

* Kludge to make `chronicle` usage in sub-modules work with `fluffy`

why:
  That `fluffy` would not run with any logging in `core_deb` is a problem
  I have known for a while. Up to now, logging was only used for debugging.

  With the current `Aristo` PR, there are cases where logging might be
  wanted but this works only if `chronicles` runs without the
  `json[dynamic]` sinks.

  So this should be re-visited.

* More of a kludge
2024-08-14 08:54:44 +00:00
web3-developer 93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
web3-developer 63d13182c1
Fluffy state bridge - Add CLI parameters and implement workers for offer gossip. (#2541)
* Use RPC batching to send offer requests and filter out duplicate offers.

* Lookup offers after gossip to check if gossip successful.

* Use multiple workers for gossiping offers.

* Update Fluffy state network logging.

* Use single RPC calls instead of batching.

* Update cli parameters.

* Fix bug in contract trie offer building.
2024-08-06 15:38:38 +08:00
web3-developer 947f629903
Fluffy State Bridge - State Gossip via Portal JSON-RPC (#2535)
* Create block offers queue and collect account preimages.

* Implement iterators to return account and storage proofs and bytecode from updatedCaches.

* Implement building offers from proofs.

* Refactor BlockDataRef type to only include required fields.

* Store block data in database.

* Improve state diff types.

* Implement start state backfill from specific block.

* Record last persisted block number in database.

* Persist account preimages in db.

* Apply state updates for DAO hard fork.

* Implement state gossip of block offers via portal JSON RPC.
2024-07-30 22:56:21 +08:00
Kim De Mey d4d8d2af64
Add validation on gossip of LC updates as per spec (#2528) 2024-07-26 18:56:32 +02:00
Kim De Mey 7e2a636717
Add basic validation for LC bootstraps + portal_bridge changes (#2527)
- Add basic validation for LC bootstrap gossip, validating either
by trusted block root (only 1) when not synced, or by comparing
with the header of the latest finality update when synced.

- Update portal_bridge beacon to also gossip bootstraps into the
network on each end of epoch.
2024-07-25 20:15:26 +02:00
Kim De Mey 942cf7e447
Fix LC start bug in portal_node start (#2524) 2024-07-25 17:10:42 +02:00
Kim De Mey 3a7f025fd9
Small cleanup/refactor on Portal history network (#2521) 2024-07-24 20:42:12 +02:00
Kim De Mey 1841fe7c4a
Remove old workaround for toString that was import-leaking in (#2520) 2024-07-24 15:33:43 +02:00
Kim De Mey 25751cade9
Fix FromAsCasing and add entry point to development Dockerfile (#2519)
* Fix FromAsCasing and add entry point to development Dockerfile

* Add copyright notice to make linter happy
2024-07-24 15:12:44 +02:00
Kim De Mey c2d9c0bc45
Remove deprecated cli options for fluffy (#2517) 2024-07-23 15:40:28 +02:00
Kim De Mey 1e1bc6623d
Fix missing changes after portal-rpc-url config change (#2516) 2024-07-23 12:46:53 +02:00
Kim De Mey 84ce0c912e
Bump Portal test vectors and re-activate skipped tests (#2515) 2024-07-23 11:59:59 +02:00
Kim De Mey 45d85aa3b5
Add HistoricalSummariesWithProof gossip to portal_bridge beacon (#2514) 2024-07-22 19:07:46 +02:00
Kim De Mey ff40c1e1a2
Update nph to 0.6.0 for fluffy and nimbus_verified_proxy (#2511) 2024-07-22 14:22:45 +02:00
Kim De Mey 5dfbebd4c0
Fix missing eraDir in path for filename in portal_bridge (#2510) 2024-07-22 11:48:21 +02:00
Kim De Mey 157fb4f1ef
Re-add toString function for proper hex logs of content keys (#2508) 2024-07-19 14:48:03 +02:00
web3-developer f9956eba59
Fluffy State Bridge - Building state using state diffs. (#2486)
* Started state bridge.

* Implement call to fetch stateDiffs using trace_replayBlockTransactions.

* Convert JSON responses to stateDiff types.

* State updates working for first few blocks.

* Correctly building state for first 200K blocks.

* Add storage of code and cleanup.

* Start state bridge refactor.

* More cleanup and fixes.

* Use RocksDb as backend for state.

* Implement transactions.

* Build RocksDb dependency when building fluffy tools.

* Move code to world state helper.

* Implement producer and consumer queue.

* Cleanup exceptions.

* Improve logging.

* Add update caches to DatabaseRef backends.
2024-07-18 17:01:40 +08:00
Kim De Mey 51cf991439
Bump ssz_serialization and use ByteList[n] + add ContentKeyByteList (#2500) 2024-07-17 17:07:27 +02:00
Kim De Mey 3bb707422b
Rework beacon block proofs to better structure (#2493)
The 3 proofs can be reworked to two proofs as we can use the
BeaconBlock directly instead of BeaconBlockHeader and
BeaconBlockBody. This is possible because the HTR of the
BeaconBlock is the same as the one of the BeaconBlockHeader.

This results in 32 bytes less as an intermediate hash can be
removed. But more importantly looks more clean and compact in
structure and code.
2024-07-17 11:32:05 +02:00
Kim De Mey 6cf57d9912
Remove duplicate logging code from fluffy (#2488)
Code was copied back when nimbus-eth2 was not a dependency.
Now that is an established dependency for many parts, lets reuse
the original code.
2024-07-15 11:41:17 +02:00
Kim De Mey 70682cd4a8
Bump NimYAML module and related changes (#2474) 2024-07-12 09:30:50 +02:00
Kim De Mey d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey 4a20756e6b
Remove unused seed_db and related code (#2471) 2024-07-10 23:02:15 +02:00
Kim De Mey 94340037bf
Set the routing table ip limits back to defaults (#2470) 2024-07-10 17:26:30 +02:00
Kim De Mey 54e3fd1a94
Move Portal wire and networks setup to new portal_node module (#2464) 2024-07-09 19:22:25 +02:00