Commit Graph

540 Commits

Author SHA1 Message Date
Kim De Mey bef4c05561
Add custom json-rpc method to initialize the trusted block root (#2805) 2024-10-30 18:32:36 +01:00
bhartnett 8d8f62bf67
Fluffy: Improve logging and add offer metrics to status logs. (#2802)
* Set offer processing logs to debug level and add offer counts to state network.

* Use metrics instead of int counters and remove from logs.

* More logging improvements. Make decoding and validation failures use error log level.

* Add protocol_id to metrics.
2024-10-30 21:17:16 +08:00
Kim De Mey e038a383c1
Add pruning of bootstraps/updates in Portal beacon network (#2779) 2024-10-30 12:51:00 +01:00
Kim De Mey 85d8ed7be1
Adjust ForkDigest dynamically in Portal beacon content tests (#2778)
Also iterates over multiple dirs now to run tests for each fork.
2024-10-27 08:34:50 +01:00
Jacek Sieka 6b2d341ebb
simplify genesis generation code (#2786)
* remove redundant abstraction
* fix misleading raises - the implementation actually swallows errors or
panics (depending on how many other layers of abstraction we penetrate
before detecting it)
2024-10-26 16:26:38 +07:00
bhartnett ca24dd915f
Fluffy: Make content request retries configurable (#2770)
* Make content request retries configurable and add to state network and history network.

* Set retries to 1. Use uint to prevent negative values. Make contentRequestRetries a debug parameter.
2024-10-25 15:15:01 +08:00
Kim De Mey 3d31911f1d
Portal beacon: Remove unneeded and confusing (fork specific) imports (#2776) 2024-10-24 20:31:17 +02:00
Jacek Sieka 5fc4c13ab1
Increase JSON-RPC limits, bump json-rpc (#2759)
* blocks can be bigger than the default 1mb when json-rpc-encoded - this
happens on sepolia for example
* json-rpc bump improves debug logging and fixes a number of bugs
* json-serialization bump fixes a crash on invalid arrays in json data

At some point, it would probably be better to compute the maximum block
size from actual block constraints, though this is somewhat tricky and
depends on gas limits etc. Until then, 16mb should be plenty.

With this, sepolia can be synced :)
2024-10-23 10:26:56 +02:00
andri lim 133387e6a7
Rework EIP-6110, EIP-7002, and EIP-7251: Pectra execution requests (#2734)
* Rework EIP-6110, EIP-7002, and EIP-7251

* Bump nimbus-eth2
2024-10-18 16:38:18 +07:00
bhartnett 47337593c9
Fluffy: Enable content cache for history network (#2745)
* Cache content after lookups in history network.

* Cleanup config in PortalProtocol.

* Use local content lookup in history network to enable using cache.
2024-10-17 16:25:53 +08:00
bhartnett 54528fb24b
Fluffy: Enable content cache for state network (#2739)
* Enable content cache for state network.

* Update state json-rpc endpoints to return local content which uses cache if enabled.

* Add content cache metrics.

* Make content cache configurable.

* Add content cache tests.
2024-10-16 21:05:39 +08:00
Kim De Mey 4191f95527
Add Docker quickstart page for Fluffy + minor docs fixes (#2744) 2024-10-16 14:56:52 +02:00
Kim De Mey b885f822d0
More clean-up since eth types refactor (#2742) 2024-10-16 14:18:02 +02:00
Kim De Mey f797d55c35
Fix, improve and update some of the fluffy docs (#2743) 2024-10-16 13:48:34 +02:00
andri lim 1126c7700d
Bump nim-eth and nimbus-eth2 (#2741)
* Bump nim-eth and nimbus-eth2

* Fix ambiguous identifier
2024-10-16 13:51:38 +07:00
Chirag Parmar 2838191c4f
replace deprecated types (#2704)
* partial commit

* fixes

* remove converters too

* revert changes on nimbus_verified_proxy

* revert changes in converter

* revert changes(re-xport) in rpc_types

* update copyright year

* replace types in other binaries

* chain config bug

* fix rebase conflict imcomplete buffer

* fix more rebase buffers

* remove ditto types and converters

* fix the tests

* update copyright year
2024-10-16 08:34:12 +07:00
Kim De Mey 9c9a41a16d
More verbose error handling for Portal stream content reading (#2737) 2024-10-14 11:53:28 +02:00
bhartnett bdbea9853e
Rename Fluffy rpc methods to match updated spec (#2735) 2024-10-14 16:46:35 +08:00
Kim De Mey 3e36f52fd7
Rename the debug docker files + add a simpler/faster one for Linux (#2731)
Also update related documentation
2024-10-13 18:07:17 +02:00
bhartnett 9db31ac17b
Fluffy: Fix state validation issue causing portal hive test failure (#2732) 2024-10-13 00:51:50 +08:00
Kim De Mey 6e114dc950
Fix JSON encoding for NodeId, no leading zeroes dropped (#2730)
Portal JSON-RPC API specifications dictate that the NodeId must be
the 32 byte identifier. We had leading zeroes being dropped in
our implementation.
2024-10-11 16:43:06 +02:00
Kim De Mey 0ebab78136
Update Grafana dashboard for Portal (#2727) 2024-10-10 20:11:04 +02:00
Kim De Mey a5f0b12bd1
Remove old portal_debug API for seeding data into the network (#2726)
This was the first API created for this, it has been superseded
by the API that makes use of era1 files and/or the portal_bridge.

Only usage was still in test_portal_testnet which has now been
altered to make use if API calls from Portal specification.
2024-10-10 16:42:57 +02:00
bhartnett 4ae87e6d19
Fluffy: Add validation and local storage of content in remaining state portal rpc methods (#2723)
* Add validation functions to be used in state portal rpc.

* Add validation to remaining state portal rpc methods.

* Lookup local content in recursiveFindContent rpc methods.

* portal_stateFindContent and portal_stateOffer no longer store in db.
2024-10-10 21:24:39 +08:00
bhartnett b13f06fcfb
Fluffy state portal rpc validation (#2719)
* Validate content key in portal_stateLocalContent.

* Add additional validation to stateStore rpc method.
2024-10-09 20:23:46 +08:00
Kim De Mey 5edb0b320f
Allow Portal beacon LC sync to start from a stored LC bootstrap (#2715)
Portal beacon LC sync can be started now from a provided trusted
block root or, in case it has been running before, from a
previously verified and stored LC bootstrap.

This required altering the the beacon db on how the bootstraps
are stored.
2024-10-09 10:21:00 +02:00
Jacek Sieka 11646ad3c4
Ordered trie (#2712)
Speed up trie computations and remove redundant ways of performing this
operation.

Co-authored-by: jangko <jangko128@gmail.com>
2024-10-09 09:44:15 +02:00
bhartnett 72ee610826
Rename Fluffy debug rpc methods (#2710)
* Rename Fluffy debug rpc methods.

* Organize portal json-rpc errors.
2024-10-08 15:21:27 +08:00
andri lim 22134fcf4c
Move premix's downloader and parser to fluffy/tools/eth_data_exporter (#2709)
* Move premix's downloader and parser to fluffy/tools/eth_data_exporter

* Run nph formatter
2024-10-08 03:07:24 +00:00
Kim De Mey a3bb5d4428
Small clean-up of the differen Portal json-rpc APIs (#2708)
Clean-up removing:
- no longer relevant comments
- unused types
- unused imports
2024-10-08 09:15:29 +08:00
Kim De Mey 833719a866
Remove usage of aliases for Hash32 such as BlockHash (#2707)
These create only confusion as if they are actual different types
and it is within their usage already clear what they are about
because of the name of the variable or the function.

They are also nowhere aliased like this in any of the Portal
specification.
2024-10-07 22:39:07 +02:00
Kim De Mey e1c942ccef
Remove unneeded w3Hash and similar calls (#2706) 2024-10-07 20:54:48 +02:00
bhartnett 344a236c31
Fluffy: Create separate portal rpc handlers for each sub-network (#2705)
* Create separate portal rpc handlers for each sub-network.

* Remove random gossip from history and state networks.

* Move shared portal rpc endpoints into common handler.

* Use enum instead of network string.
2024-10-07 16:33:02 +02:00
Kim De Mey 778c1d589f
Adapt portal_*Offer method to the latest specification (#2689) 2024-10-07 10:49:04 +02:00
tersec 845f3276e3
bump nimbus-build-system to use Nim v2.0.10 (#2684)
* bump nimbus-build-system to use Nim v2.0.10

* 2.0.10 fixes

* fluffy linting

* make trivial change which should trigger whole-nimbus+fluffy rebuild/ci

* Nim v2.0.10 chronicles.error/macros.error ambiguity workaround

* another contentType enum specifier

* fluffy linting
2024-10-06 12:15:54 +00:00
Kim De Mey 91e18caf84
Fix Portal beacon statusLogLoop that was not running (#2702) 2024-10-05 13:39:42 +02:00
Kim De Mey dbe3393f5c
Fix eth/common & web3 related deprecation warnings for fluffy (#2698)
* Fix eth/common & web3 related deprecation warnings for fluffy

This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.

It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.

There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.

* Address review comments

* Remove two more unused eth/common imports
2024-10-04 23:21:26 +02:00
Jacek Sieka 08ffb3161c
Use eth/common transaction signature utilities (#2696)
* Use eth/common transaction signature utilities

* bump

* bump

* bump

* bump

* bump

* bump
2024-10-04 16:34:31 +02:00
bhartnett 56724536a4
Fluffy State Bridge: Improvements to websocket rpc connections (#2694)
Handle reconnect for websocket rpc client and create rpc client instance per worker in order to improve websocket throughput. (#2694)
2024-10-04 14:17:18 +08:00
Kim De Mey 0b81a8c783
Adjust error code on portal_*TraceRecursiveFindContent (#2687) 2024-10-03 14:25:49 +02:00
Kim De Mey 764c950135
Remove PREFER_BLST_SHA256 disable, allowing for BLST SHA256 in SSZ (#2686) 2024-10-03 14:25:30 +02:00
bhartnett 9192aa13ed
Fluffy state bridge updates (#2683)
* Use Json decode when handling portal rpc client error.

* Remove preimages cache.

* Support disabling state gossip in order to just build state.
2024-10-03 10:57:19 +08:00
Jacek Sieka a03bb56bec
update web3 types to common eth (#2674) 2024-10-02 18:22:35 +02:00
bhartnett 44fea2348b
Fluffy Portal RPC Client (#2649)
* Portal rpc client implementation.
2024-10-02 12:23:13 +08:00
Jacek Sieka 219b22b1f5
Versioned hash32 (#2672) 2024-10-01 19:40:37 +02:00
Jacek Sieka c210885b73
eth: bump to new types (#2660)
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
2024-09-29 14:37:09 +02:00
Kim De Mey 438e183586
Rename to HistoricalHashesAccumulator as per Portal spec (#2663) 2024-09-27 22:25:26 +02:00
Bhaskar Metiya b0a4a9ef84
Fix multiple instances running from same dataDir (#2647)
* Fix multiple instances running from same dataDir

* Add exclusive lock on lock file

* Unlock lock file on process exit

* Fix minor issues in lock file implementation
2024-09-26 10:09:34 +02:00
Kim De Mey 5e3f3db165
Remove client-side json-rpc calls that were removed server-side (#2659) 2024-09-25 22:35:19 +02:00
Kim De Mey dd5bb695ed
Tune revalidateMax + defaultMaxGossipNodes values in wire protocol (#2658)
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.

The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
2024-09-25 17:38:33 +02:00