492 Commits

Author SHA1 Message Date
Kim De Mey
5e3f3db165
Remove client-side json-rpc calls that were removed server-side (#2659) 2024-09-25 22:35:19 +02:00
Kim De Mey
dd5bb695ed
Tune revalidateMax + defaultMaxGossipNodes values in wire protocol (#2658)
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.

The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
2024-09-25 17:38:33 +02:00
web3-developer
69d58e8215
Fluffy: Configure RPC APIs via CLI parameter (#2657)
* Support RPC API namespaces as cli parameter.

* Fluffy now uses rpcFlags on startup.

* Update testnet script to enable all RPC APIs.

* Update Fluffy book and move web3 call into eth calls.
2024-09-25 22:44:46 +08:00
Kim De Mey
3820b15f28
Fix bug that would cause no nodes to be selected for revalidation (#2656)
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely

Also add metrics for the amount of offers done via POKE mechanism.
2024-09-25 11:30:42 +02:00
web3-developer
cb69723ff3
Fluffy: Make number of nodes to gossip content to configurable (#2653) 2024-09-25 15:15:20 +08:00
web3-developer
b1dc1578f0
Move web3 rpc handler into eth handler. Create debug rpc handler and move existing debug endpoints. (#2655) 2024-09-25 11:28:45 +08:00
Kim De Mey
e603952df3
Refactor portal test directory structure (#2650) 2024-09-24 13:07:20 +02:00
Kim De Mey
0fb9581b96
Refactor and apply renaming to post merge block proofs (#2648) 2024-09-23 18:56:28 +02:00
web3-developer
07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
Kim De Mey
c77c6239b0
Use discv5's maxDiscv5TalkRespPayload const for payload size calc (#2643) 2024-09-19 20:30:10 +02:00
web3-developer
a9ad10cadc
Fluffy state network now enabled by default and improve status logs (#2640)
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
2024-09-19 21:38:49 +08:00
web3-developer
ea74e035ab
Add WebSocket support to Fluffy. (#2636)
* Add WebSocket support to Fluffy.

* Support websocket compression.

* Create setupRpcServer closure to remove code duplication.
2024-09-18 21:27:24 +08:00
web3-developer
0ee8e61a3a
Remove RpcProxy from Fluffy and use RpcHttpServer instead. (#2634)
* Remove RpcProxy from Fluffy and use RpcHttpServer instead.

* Cleanup test.
2024-09-18 15:46:50 +08:00
web3-developer
c652f5efc2
Fluffy state bridge docs (#2632)
* Don't proxy implemented state JSON-RPC endpoints. Fix minor issue in testnet script.

* Add docs covering usage of the Fluffy state bridge.
2024-09-18 11:56:20 +08:00
web3-developer
d17ddacf39
More improvements to the Fluffy local testnet script (#2624)
* More improvements to the Fluffy local testnet script.
2024-09-13 20:40:48 +08:00
Kim De Mey
d96196d01d
Add support for BlockHeader by number in portal_bridge (#2621) 2024-09-13 13:41:34 +02:00
web3-developer
b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer
9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey
45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
Kim De Mey
fd3475ea3e
Add a brief doc page on how to add documentation (#2617) 2024-09-12 10:50:45 +02:00
Kim De Mey
4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
web3-developer
3a73b948c6
Fluffy testnet script updates (#2615)
* Update local testnet script to support additional options.

* Increase state bridge retry sleep times.

* Fix run fluffy testnet tests in CI.
2024-09-12 16:18:24 +08:00
tersec
1b173d420d
small cleanups (#2598)
* small cleanups

* stop hiding ConvFromXtoItselfNotNeeded hints

* lowmem optimization flag is no-op
2024-09-10 05:24:45 +00:00
Kim De Mey
0869a27462
Move inRange check to the Portal protocol + related simplifications (#2602) 2024-09-09 17:52:11 +02:00
Kim De Mey
cb94dd0c5b
Small cleanup of the ContentDB store handler (#2597) 2024-09-07 15:03:13 +02:00
Kim De Mey
e919f57902
Remove old header network code as this no longer exists in specs (#2596) 2024-09-06 18:00:58 +02:00
Kim De Mey
8a19118ddf
Fix initial radius bug due to uninitialized localId (#2595) 2024-09-06 11:45:27 +02:00
Kim De Mey
0a80a3bb25
Reverse calculate the radius at node restart (#2593)
This avoid restarting the node always with a full radius, which
causes the node the be bombarded with offers which it later has
to delete anyhow.

In order to implement this functionality, several changes were
made as the radius needed to move from the Portal wire protocol
current location to the contentDB and beaconDB, which is
conceptually more correct anyhow.

So radius is now part of the database objects and a handler is
used in the portal wire protocol to access its value.
2024-09-05 18:31:55 +02:00
web3-developer
139b35a80a
Fluffy State Bridge: State building fixes (#2589)
* Fix bug where code was not correctly being updated during contract creation.

* Delete value from update cache during db delete.
2024-09-03 21:05:38 +08:00
web3-developer
ee6a7e8259
Fluffy state endpoint improvements (#2580)
* Return default values when account, slot or code doesn't exist.

* Handle case when storage doesn't exist due to account not existing or being a non contract account.
2024-08-28 16:27:36 +08:00
web3-developer
fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer
507a9e71df
Fluffy state network fixes and improvements (#2576)
* Cleanup tests.

* Improve offer sort function in state bridge.

* Verify nibble prefix for leaf and extension nodes during lookup.
2024-08-23 15:46:23 +08:00
web3-developer
60c9b2c00d
Fluffy State Bridge: Fetch portal client nodeId on startup and sort offers by distance from nodeId (#2570)
* Fetch portal client nodeId on startup and sort offers by distance from nodeId.

* Fix logging.

* Improve error handling at startup.
2024-08-19 20:49:07 +08:00
web3-developer
c8d34eba9b
Bump portal-spec-tests and update Fluffy state tests (#2568)
* Bump portal-spec-tests.

* Update state network tests to use block_header instead of state_root.
2024-08-19 14:45:54 +08:00
Jordan Hrycaj
cbe5131927
Simplify aristo tree deletion functionality (#2563)
* Cleaning up, removing cruft and debugging statements

* Make `aristo_delta` fluffy compatible

why:
  A sub-module that uses `chronicles` must import all possible
  modules used by a parent module that imports the sub-module.

* update TODO
2024-08-14 12:09:30 +00:00
Jordan Hrycaj
ce713d95fc
Aristo lazily delete larger subtrees (#2560)
* Extract sub-tree deletion functions into separate sub-modules

* Move/rename `aristo_desc.accLruSize` => `aristo_constants.ACC_LRU_SIZE`

* Lazily delete sub-trees

why:
  This gives some control of the memory used to keep the deleted vertices
  in the cached layers. For larger sub-trees, keys and vertices might be
  on the persistent backend to a large extend. This would pull an amount
  of extra information from the backend into the cached layer.

  For lazy deleting it is enough to remember sub-trees by a small set of
  (at most 16) sub-roots to be processed when storing persistent data.
  Marking the tree root deleted immediately allows to let most of the code
  base work as before.

* Comments and cosmetics

* No need to import all for `Aristo` here

* Kludge to make `chronicle` usage in sub-modules work with `fluffy`

why:
  That `fluffy` would not run with any logging in `core_deb` is a problem
  I have known for a while. Up to now, logging was only used for debugging.

  With the current `Aristo` PR, there are cases where logging might be
  wanted but this works only if `chronicles` runs without the
  `json[dynamic]` sinks.

  So this should be re-visited.

* More of a kludge
2024-08-14 08:54:44 +00:00
web3-developer
93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
web3-developer
63d13182c1
Fluffy state bridge - Add CLI parameters and implement workers for offer gossip. (#2541)
* Use RPC batching to send offer requests and filter out duplicate offers.

* Lookup offers after gossip to check if gossip successful.

* Use multiple workers for gossiping offers.

* Update Fluffy state network logging.

* Use single RPC calls instead of batching.

* Update cli parameters.

* Fix bug in contract trie offer building.
2024-08-06 15:38:38 +08:00
web3-developer
947f629903
Fluffy State Bridge - State Gossip via Portal JSON-RPC (#2535)
* Create block offers queue and collect account preimages.

* Implement iterators to return account and storage proofs and bytecode from updatedCaches.

* Implement building offers from proofs.

* Refactor BlockDataRef type to only include required fields.

* Store block data in database.

* Improve state diff types.

* Implement start state backfill from specific block.

* Record last persisted block number in database.

* Persist account preimages in db.

* Apply state updates for DAO hard fork.

* Implement state gossip of block offers via portal JSON RPC.
2024-07-30 22:56:21 +08:00
Kim De Mey
d4d8d2af64
Add validation on gossip of LC updates as per spec (#2528) 2024-07-26 18:56:32 +02:00
Kim De Mey
7e2a636717
Add basic validation for LC bootstraps + portal_bridge changes (#2527)
- Add basic validation for LC bootstrap gossip, validating either
by trusted block root (only 1) when not synced, or by comparing
with the header of the latest finality update when synced.

- Update portal_bridge beacon to also gossip bootstraps into the
network on each end of epoch.
2024-07-25 20:15:26 +02:00
Kim De Mey
942cf7e447
Fix LC start bug in portal_node start (#2524) 2024-07-25 17:10:42 +02:00
Kim De Mey
3a7f025fd9
Small cleanup/refactor on Portal history network (#2521) 2024-07-24 20:42:12 +02:00
Kim De Mey
1841fe7c4a
Remove old workaround for toString that was import-leaking in (#2520) 2024-07-24 15:33:43 +02:00
Kim De Mey
25751cade9
Fix FromAsCasing and add entry point to development Dockerfile (#2519)
* Fix FromAsCasing and add entry point to development Dockerfile

* Add copyright notice to make linter happy
2024-07-24 15:12:44 +02:00
Kim De Mey
c2d9c0bc45
Remove deprecated cli options for fluffy (#2517) 2024-07-23 15:40:28 +02:00
Kim De Mey
1e1bc6623d
Fix missing changes after portal-rpc-url config change (#2516) 2024-07-23 12:46:53 +02:00
Kim De Mey
84ce0c912e
Bump Portal test vectors and re-activate skipped tests (#2515) 2024-07-23 11:59:59 +02:00
Kim De Mey
45d85aa3b5
Add HistoricalSummariesWithProof gossip to portal_bridge beacon (#2514) 2024-07-22 19:07:46 +02:00
Kim De Mey
ff40c1e1a2
Update nph to 0.6.0 for fluffy and nimbus_verified_proxy (#2511) 2024-07-22 14:22:45 +02:00