300 Commits

Author SHA1 Message Date
Jacek Sieka
11646ad3c4
Ordered trie (#2712)
Speed up trie computations and remove redundant ways of performing this
operation.

Co-authored-by: jangko <jangko128@gmail.com>
2024-10-09 09:44:15 +02:00
Kim De Mey
833719a866
Remove usage of aliases for Hash32 such as BlockHash (#2707)
These create only confusion as if they are actual different types
and it is within their usage already clear what they are about
because of the name of the variable or the function.

They are also nowhere aliased like this in any of the Portal
specification.
2024-10-07 22:39:07 +02:00
Kim De Mey
778c1d589f
Adapt portal_*Offer method to the latest specification (#2689) 2024-10-07 10:49:04 +02:00
tersec
845f3276e3
bump nimbus-build-system to use Nim v2.0.10 (#2684)
* bump nimbus-build-system to use Nim v2.0.10

* 2.0.10 fixes

* fluffy linting

* make trivial change which should trigger whole-nimbus+fluffy rebuild/ci

* Nim v2.0.10 chronicles.error/macros.error ambiguity workaround

* another contentType enum specifier

* fluffy linting
2024-10-06 12:15:54 +00:00
Kim De Mey
91e18caf84
Fix Portal beacon statusLogLoop that was not running (#2702) 2024-10-05 13:39:42 +02:00
Kim De Mey
dbe3393f5c
Fix eth/common & web3 related deprecation warnings for fluffy (#2698)
* Fix eth/common & web3 related deprecation warnings for fluffy

This commit uses the new types in the new eth/common/ structure
to remove deprecation warnings.

It is however more than just a mass replace as also all places
where eth/common or eth/common/eth_types or eth/common/eth_types_rlp
got imported have been revised and adjusted to a better per submodule
based import.

There are still a bunch of toMDigest deprecation warnings but that
convertor is not needed for fluffy code anymore so in theory it
should not be used (bug?). It seems to still get imported via export
leaks ffrom imported nimbus code I think.

* Address review comments

* Remove two more unused eth/common imports
2024-10-04 23:21:26 +02:00
Jacek Sieka
c210885b73
eth: bump to new types (#2660)
This is a minimal set of changes to make things work with the new types
in nim-eth - this is the minimal PR that merely resolves
incompatibilities while the full change set would include more cleanup
and migration.
2024-09-29 14:37:09 +02:00
Kim De Mey
438e183586
Rename to HistoricalHashesAccumulator as per Portal spec (#2663) 2024-09-27 22:25:26 +02:00
Kim De Mey
dd5bb695ed
Tune revalidateMax + defaultMaxGossipNodes values in wire protocol (#2658)
The revalidateMax value is lowered to have a quicker ramp up of
the radiusCache + to keep it healthier.

The defaultMaxGossipNodes value is lowered because with the
current value a Nodes lookup is triggered almost always.
This value is dependant on the content replication value. This
is dependant on the network (and subnetwork) because of the amount
of nodes and their radius/storage capacity.
2024-09-25 17:38:33 +02:00
Kim De Mey
3820b15f28
Fix bug that would cause no nodes to be selected for revalidation (#2656)
This bug would have as effect that our radius cache would not get
filled by any outgoing pings, causing:
- Node lookups to always be occurring on NH gossip
- POKEs to much more rarely

Also add metrics for the amount of offers done via POKE mechanism.
2024-09-25 11:30:42 +02:00
web3-developer
cb69723ff3
Fluffy: Make number of nodes to gossip content to configurable (#2653) 2024-09-25 15:15:20 +08:00
Kim De Mey
e603952df3
Refactor portal test directory structure (#2650) 2024-09-24 13:07:20 +02:00
Kim De Mey
0fb9581b96
Refactor and apply renaming to post merge block proofs (#2648) 2024-09-23 18:56:28 +02:00
web3-developer
07193968ca
Implement graceful shutdown in Fluffy (#2645)
* Make stop functions wait for completion before return.

* Implement graceful shutdown.

* Shutdown rpc and metric servers if enabled.

* Move metrics and rpc servers out of PortalNode.
2024-09-20 20:54:36 +08:00
Kim De Mey
c77c6239b0
Use discv5's maxDiscv5TalkRespPayload const for payload size calc (#2643) 2024-09-19 20:30:10 +02:00
web3-developer
a9ad10cadc
Fluffy state network now enabled by default and improve status logs (#2640)
* Enable state network by default. Create status log loop for state and beacon networks. Create status log loop for portal node. Implement stop functions.
2024-09-19 21:38:49 +08:00
web3-developer
b11701c75a
Update state JSON-RPC endpoints to directly lookup by blockNumber and skip lookup by blockHash. (#2623) 2024-09-13 14:51:16 +08:00
web3-developer
9c1594b417
Implement eth_getProof JSON-RPC API in Fluffy (#2622)
* Improve state endpoint genesis test and cover cases when accounts, code and slots doesn't exist.

* Refactor state endpoints to support returning partial proofs.

* Implement getProofs in state endpoints.

* Add tests for getProofs and improve code.

* Implement eth_getProof JSON-RPC api in Fluffy.
2024-09-13 11:46:35 +08:00
Kim De Mey
45867bc214
Add BlockHeader by number to Portal history network (#2620)
- Add new content + content key functionality for header by number
- Remove EpochRecords from the network
- Add pruning call for the EpochRecords + required deprecated
functionality
- Adjust getBlock and getBlockHashByNumber to make use of the
new functionality instead
- Delete content_verifier as it was only verifying the now
deprecated EpochRecord
2024-09-12 19:03:58 +02:00
Kim De Mey
4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
Kim De Mey
0869a27462
Move inRange check to the Portal protocol + related simplifications (#2602) 2024-09-09 17:52:11 +02:00
Kim De Mey
e919f57902
Remove old header network code as this no longer exists in specs (#2596) 2024-09-06 18:00:58 +02:00
Kim De Mey
0a80a3bb25
Reverse calculate the radius at node restart (#2593)
This avoid restarting the node always with a full radius, which
causes the node the be bombarded with offers which it later has
to delete anyhow.

In order to implement this functionality, several changes were
made as the radius needed to move from the Portal wire protocol
current location to the contentDB and beaconDB, which is
conceptually more correct anyhow.

So radius is now part of the database objects and a handler is
used in the portal wire protocol to access its value.
2024-09-05 18:31:55 +02:00
web3-developer
ee6a7e8259
Fluffy state endpoint improvements (#2580)
* Return default values when account, slot or code doesn't exist.

* Handle case when storage doesn't exist due to account not existing or being a non contract account.
2024-08-28 16:27:36 +08:00
web3-developer
fa59898388
Fluffy state debug endpoints (#2578)
* Add debug endpoints that support looking up state by state root.

* Test lookup by state root endpoints.
2024-08-27 20:35:27 +08:00
web3-developer
507a9e71df
Fluffy state network fixes and improvements (#2576)
* Cleanup tests.

* Improve offer sort function in state bridge.

* Verify nibble prefix for leaf and extension nodes during lookup.
2024-08-23 15:46:23 +08:00
web3-developer
93a160b569
Update Fluffy State Network to match Portal spec addressHash change (#2548)
* Update state network to use addressHash instead of address in contract trie and contract code content keys.

* Fix path calculation bug in getParent when working with extension nodes.

* Bump portal spec tests repo.

* Finish updating tests due to portal test vector changes.

* Update Fluffy state bridge to use addressHash.

* Update Fluffy book with correct commands for running portal hive tests.
2024-08-08 00:01:30 +08:00
web3-developer
63d13182c1
Fluffy state bridge - Add CLI parameters and implement workers for offer gossip. (#2541)
* Use RPC batching to send offer requests and filter out duplicate offers.

* Lookup offers after gossip to check if gossip successful.

* Use multiple workers for gossiping offers.

* Update Fluffy state network logging.

* Use single RPC calls instead of batching.

* Update cli parameters.

* Fix bug in contract trie offer building.
2024-08-06 15:38:38 +08:00
web3-developer
947f629903
Fluffy State Bridge - State Gossip via Portal JSON-RPC (#2535)
* Create block offers queue and collect account preimages.

* Implement iterators to return account and storage proofs and bytecode from updatedCaches.

* Implement building offers from proofs.

* Refactor BlockDataRef type to only include required fields.

* Store block data in database.

* Improve state diff types.

* Implement start state backfill from specific block.

* Record last persisted block number in database.

* Persist account preimages in db.

* Apply state updates for DAO hard fork.

* Implement state gossip of block offers via portal JSON RPC.
2024-07-30 22:56:21 +08:00
Kim De Mey
d4d8d2af64
Add validation on gossip of LC updates as per spec (#2528) 2024-07-26 18:56:32 +02:00
Kim De Mey
7e2a636717
Add basic validation for LC bootstraps + portal_bridge changes (#2527)
- Add basic validation for LC bootstrap gossip, validating either
by trusted block root (only 1) when not synced, or by comparing
with the header of the latest finality update when synced.

- Update portal_bridge beacon to also gossip bootstraps into the
network on each end of epoch.
2024-07-25 20:15:26 +02:00
Kim De Mey
3a7f025fd9
Small cleanup/refactor on Portal history network (#2521) 2024-07-24 20:42:12 +02:00
Kim De Mey
1841fe7c4a
Remove old workaround for toString that was import-leaking in (#2520) 2024-07-24 15:33:43 +02:00
Kim De Mey
45d85aa3b5
Add HistoricalSummariesWithProof gossip to portal_bridge beacon (#2514) 2024-07-22 19:07:46 +02:00
Kim De Mey
ff40c1e1a2
Update nph to 0.6.0 for fluffy and nimbus_verified_proxy (#2511) 2024-07-22 14:22:45 +02:00
Kim De Mey
51cf991439
Bump ssz_serialization and use ByteList[n] + add ContentKeyByteList (#2500) 2024-07-17 17:07:27 +02:00
Kim De Mey
3bb707422b
Rework beacon block proofs to better structure (#2493)
The 3 proofs can be reworked to two proofs as we can use the
BeaconBlock directly instead of BeaconBlockHeader and
BeaconBlockBody. This is possible because the HTR of the
BeaconBlock is the same as the one of the BeaconBlockHeader.

This results in 32 bytes less as an intermediate hash can be
removed. But more importantly looks more clean and compact in
structure and code.
2024-07-17 11:32:05 +02:00
Kim De Mey
d996e60347
Rename to EpochRecord and other accumulator spec changes (#2473)
- EpochAccumulator got renamed to EpochRecord
- MasterAccumulator is not HistoricalHashesAccumulator
- The List size for the accumulator got a different maximum which
also result in a different encoding and HTR
2024-07-11 17:42:45 +02:00
Kim De Mey
4a20756e6b
Remove unused seed_db and related code (#2471) 2024-07-10 23:02:15 +02:00
Kim De Mey
94340037bf
Set the routing table ip limits back to defaults (#2470) 2024-07-10 17:26:30 +02:00
Kim De Mey
54e3fd1a94
Move Portal wire and networks setup to new portal_node module (#2464) 2024-07-09 19:22:25 +02:00
andri lim
d9e502bbc5
Bump web3/kzg4844/nimbus-eth2 and related fixes (#2446) 2024-07-04 05:41:32 +00:00
Kim De Mey
87d090afa6
Refactor history/beacon content types to seperate keys and values (#2438)
Refactor content types, to have similar structure as for state
network.
Amd some clean-up of comments/TODOs/spec links.
2024-07-02 15:48:49 +02:00
Kim De Mey
ae094692fb
Bump nim-eth for ENR changes and related clean-up (#2422) 2024-06-27 17:59:08 +02:00
web3-developer
09946c9958
Fluffy state network tests and logging improvements (#2402)
* Improve tests by waiting for content to get into db.

* Improve state network logging.
2024-06-20 23:41:34 +08:00
web3-developer
bd2ca07da6
Update Fluffy data directory location based on selected network. (#2400) 2024-06-20 10:48:45 +08:00
web3-developer
0b8363764d
More state endpoint tests (#2399) 2024-06-20 00:21:23 +08:00
Miran
ea0d18424a
use Nim 2.0.6 (#2384)
* use Nim 2.0.6

* Fixes for nim 2.0.6

* Workaround nim 2.0 array indexing issue

* Remove excess gcsafe pragma

* Oops, fix recursive template

* Fix imports

* Fluffy nph linting

---------

Co-authored-by: jangko <jangko128@gmail.com>
Co-authored-by: tersec <tersec@users.noreply.github.com>
2024-06-19 01:27:54 +00:00
web3-developer
e3d14bd921
Fluffy portal testnet support (#2383)
* Bump portal-mainnet repo.

* Update command line arguments and parsing on startup.

* Read in angelfood bootstrap nodes and update Fluffy guide.

* Configure subnetwork protocol ids.
2024-06-18 15:32:57 +08:00
web3-developer
1377f93d50
Implement Fluffy JSON-RPC endpoints. (#2364)
* Implement eth_getBalance, eth_getTransactionCount, eth_getStorageAt and eth_getCode JSON-RPCs.

* Fixes.
2024-06-15 01:03:18 +08:00