Commit Graph

531 Commits

Author SHA1 Message Date
bhartnett 47337593c9
Fluffy: Enable content cache for history network (#2745)
* Cache content after lookups in history network.

* Cleanup config in PortalProtocol.

* Use local content lookup in history network to enable using cache.
2024-10-17 16:25:53 +08:00
bhartnett 54528fb24b
Fluffy: Enable content cache for state network (#2739)
* Enable content cache for state network.

* Update state json-rpc endpoints to return local content which uses cache if enabled.

* Add content cache metrics.

* Make content cache configurable.

* Add content cache tests.
2024-10-16 21:05:39 +08:00
Kim De Mey 4191f95527
Add Docker quickstart page for Fluffy + minor docs fixes (#2744) 2024-10-16 14:56:52 +02:00
Kim De Mey b885f822d0
More clean-up since eth types refactor (#2742) 2024-10-16 14:18:02 +02:00
Kim De Mey f797d55c35
Fix, improve and update some of the fluffy docs (#2743) 2024-10-16 13:48:34 +02:00
andri lim 1126c7700d
Bump nim-eth and nimbus-eth2 (#2741)
* Bump nim-eth and nimbus-eth2

* Fix ambiguous identifier
2024-10-16 13:51:38 +07:00
Chirag Parmar 2838191c4f
replace deprecated types (#2704)
* partial commit

* fixes

* remove converters too

* revert changes on nimbus_verified_proxy

* revert changes in converter

* revert changes(re-xport) in rpc_types

* update copyright year

* replace types in other binaries

* chain config bug

* fix rebase conflict imcomplete buffer

* fix more rebase buffers

* remove ditto types and converters

* fix the tests

* update copyright year
2024-10-16 08:34:12 +07:00
Kim De Mey 9c9a41a16d
More verbose error handling for Portal stream content reading (#2737) 2024-10-14 11:53:28 +02:00
bhartnett bdbea9853e
Rename Fluffy rpc methods to match updated spec (#2735) 2024-10-14 16:46:35 +08:00
Kim De Mey 3e36f52fd7
Rename the debug docker files + add a simpler/faster one for Linux (#2731)
Also update related documentation
2024-10-13 18:07:17 +02:00
bhartnett 9db31ac17b
Fluffy: Fix state validation issue causing portal hive test failure (#2732) 2024-10-13 00:51:50 +08:00
Kim De Mey 6e114dc950
Fix JSON encoding for NodeId, no leading zeroes dropped (#2730)
Portal JSON-RPC API specifications dictate that the NodeId must be
the 32 byte identifier. We had leading zeroes being dropped in
our implementation.
2024-10-11 16:43:06 +02:00
Kim De Mey 0ebab78136
Update Grafana dashboard for Portal (#2727) 2024-10-10 20:11:04 +02:00
Kim De Mey a5f0b12bd1
Remove old portal_debug API for seeding data into the network (#2726)
This was the first API created for this, it has been superseded
by the API that makes use of era1 files and/or the portal_bridge.

Only usage was still in test_portal_testnet which has now been
altered to make use if API calls from Portal specification.
2024-10-10 16:42:57 +02:00
bhartnett 4ae87e6d19
Fluffy: Add validation and local storage of content in remaining state portal rpc methods (#2723)
* Add validation functions to be used in state portal rpc.

* Add validation to remaining state portal rpc methods.

* Lookup local content in recursiveFindContent rpc methods.

* portal_stateFindContent and portal_stateOffer no longer store in db.
2024-10-10 21:24:39 +08:00
bhartnett b13f06fcfb
Fluffy state portal rpc validation (#2719)
* Validate content key in portal_stateLocalContent.

* Add additional validation to stateStore rpc method.
2024-10-09 20:23:46 +08:00
Kim De Mey 5edb0b320f
Allow Portal beacon LC sync to start from a stored LC bootstrap (#2715)
Portal beacon LC sync can be started now from a provided trusted
block root or, in case it has been running before, from a
previously verified and stored LC bootstrap.

This required altering the the beacon db on how the bootstraps
are stored.
2024-10-09 10:21:00 +02:00
Jacek Sieka 11646ad3c4
Ordered trie (#2712)
Speed up trie computations and remove redundant ways of performing this
operation.

Co-authored-by: jangko <jangko128@gmail.com>
2024-10-09 09:44:15 +02:00
bhartnett 72ee610826
Rename Fluffy debug rpc methods (#2710)
* Rename Fluffy debug rpc methods.

* Organize portal json-rpc errors.
2024-10-08 15:21:27 +08:00
andri lim 22134fcf4c
Move premix's downloader and parser to fluffy/tools/eth_data_exporter (#2709)
* Move premix's downloader and parser to fluffy/tools/eth_data_exporter

* Run nph formatter
2024-10-08 03:07:24 +00:00
Kim De Mey a3bb5d4428
Small clean-up of the differen Portal json-rpc APIs (#2708)
Clean-up removing:
- no longer relevant comments
- unused types
- unused imports
2024-10-08 09:15:29 +08:00
Kim De Mey 833719a866
Remove usage of aliases for Hash32 such as BlockHash (#2707)
These create only confusion as if they are actual different types
and it is within their usage already clear what they are about
because of the name of the variable or the function.

They are also nowhere aliased like this in any of the Portal
specification.
2024-10-07 22:39:07 +02:00
Kim De Mey e1c942ccef
Remove unneeded w3Hash and similar calls (#2706) 2024-10-07 20:54:48 +02:00
bhartnett 344a236c31
Fluffy: Create separate portal rpc handlers for each sub-network (#2705)
* Create separate portal rpc handlers for each sub-network.

* Remove random gossip from history and state networks.

* Move shared portal rpc endpoints into common handler.

* Use enum instead of network string.
2024-10-07 16:33:02 +02:00
Kim De Mey 778c1d589f
Adapt portal_*Offer method to the latest specification (#2689) 2024-10-07 10:49:04 +02:00
tersec 845f3276e3
bump nimbus-build-system to use Nim v2.0.10 (#2684)
* bump nimbus-build-system to use Nim v2.0.10

* 2.0.10 fixes

* fluffy linting

* make trivial change which should trigger whole-nimbus+fluffy rebuild/ci

* Nim v2.0.10 chronicles.error/macros.error ambiguity workaround

* another contentType enum specifier

* fluffy linting
2024-10-06 12:15:54 +00:00
Kim De Mey 91e18caf84
Fix Portal beacon statusLogLoop that was not running (#2702) 2024-10-05 13:39:42 +02:00
Kim De Mey dbe3393f5c
Fix eth/common & web3 related deprecation warnings for fluffy (#2698)
* Fix eth/common & web3 related deprecation warnings for fluffy

This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.

It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.

There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.

* Address review comments

* Remove two more unused eth/common imports
2024-10-04 23:21:26 +02:00
Jacek Sieka 08ffb3161c
Use eth/common transaction signature utilities (#2696)
* Use eth/common transaction signature utilities

* bump

* bump

* bump

* bump

* bump

* bump
2024-10-04 16:34:31 +02:00
bhartnett 56724536a4
Fluffy State Bridge: Improvements to websocket rpc connections (#2694)
Handle reconnect for websocket rpc client and create rpc client instance per worker in order to improve websocket throughput. (#2694)
2024-10-04 14:17:18 +08:00
Kim De Mey 0b81a8c783
Adjust error code on portal_*TraceRecursiveFindContent (#2687) 2024-10-03 14:25:49 +02:00
Kim De Mey 764c950135
Remove PREFER_BLST_SHA256 disable, allowing for BLST SHA256 in SSZ (#2686) 2024-10-03 14:25:30 +02:00
bhartnett 9192aa13ed
Fluffy state bridge updates (#2683)
* Use Json decode when handling portal rpc client error.

* Remove preimages cache.

* Support disabling state gossip in order to just build state.
2024-10-03 10:57:19 +08:00
Jacek Sieka a03bb56bec
update web3 types to common eth (#2674) 2024-10-02 18:22:35 +02:00
bhartnett 44fea2348b
Fluffy Portal RPC Client (#2649)
* Portal rpc client implementation.
2024-10-02 12:23:13 +08:00
Jacek Sieka 219b22b1f5
Versioned hash32 (#2672) 2024-10-01 19:40:37 +02:00
Jacek Sieka c210885b73
eth: bump to new types (#2660)
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
2024-09-29 14:37:09 +02:00
Kim De Mey 438e183586
Rename to HistoricalHashesAccumulator as per Portal spec (#2663) 2024-09-27 22:25:26 +02:00
Bhaskar Metiya b0a4a9ef84
Fix multiple instances running from same dataDir (#2647)
* Fix multiple instances running from same dataDir

* Add exclusive lock on lock file

* Unlock lock file on process exit

* Fix minor issues in lock file implementation
2024-09-26 10:09:34 +02:00
Kim De Mey 5e3f3db165
Remove client-side json-rpc calls that were removed server-side (#2659) 2024-09-25 22:35:19 +02:00
Kim De Mey dd5bb695ed
Tune revalidateMax + defaultMaxGossipNodes values in wire protocol (#2658)
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.

The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
2024-09-25 17:38:33 +02:00
web3-developer 69d58e8215
Fluffy: Configure RPC APIs via CLI parameter (#2657)
* Support RPC API namespaces as cli parameter.

* Fluffy now uses rpcFlags on startup.

* Update testnet script to enable all RPC APIs.

* Update Fluffy book and move web3 call into eth calls.
2024-09-25 22:44:46 +08:00
Kim De Mey 3820b15f28
Fix bug that would cause no nodes to be selected for revalidation (#2656)
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely

Also add metrics for the amount of offers done via POKE mechanism.
2024-09-25 11:30:42 +02:00
web3-developer cb69723ff3
Fluffy: Make number of nodes to gossip content to configurable (#2653) 2024-09-25 15:15:20 +08:00
web3-developer b1dc1578f0
Move web3 rpc handler into eth handler. Create debug rpc handler and move existing debug endpoints. (#2655) 2024-09-25 11:28:45 +08:00
Kim De Mey e603952df3
Refactor portal test directory structure (#2650) 2024-09-24 13:07:20 +02:00
Kim De Mey 0fb9581b96
Refactor and apply renaming to post merge block proofs (#2648) 2024-09-23 18:56:28 +02:00
web3-developer 07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
Kim De Mey c77c6239b0
Use discv5's maxDiscv5TalkRespPayload const for payload size calc (#2643) 2024-09-19 20:30:10 +02:00
web3-developer a9ad10cadc
Fluffy state network now enabled by default and improve status logs (#2640)
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
2024-09-19 21:38:49 +08:00