Commit Graph

294 Commits

Author SHA1 Message Date
web3-developer c72d6aa5d6
Implementation of procs to support Fluffy state JSON-RPC endpoints. (#2318)
* Started implementation of state endpoints.

* Add rpc calls and server stubs.

* Initial implementation of getAccountProof and getStorageProof.

* Refactor validation to use toAccount utils functions.

* Add state endpoints tests.
2024-06-10 18:47:09 +08:00
Jacek Sieka f6be4bd0ec
avoid initTable (#2328)
`initTable` is obsolete since nim 0.19 and can introduce significant
memory overhead while providing no benefit (since the table will be
grown to the default initial size on first use anyway).

In particular, aristo layers will not necessarily use all tables they
initialize, for exampe when many empty accounts are being created.
2024-06-10 11:05:30 +02:00
Kim De Mey d814d84b9b
Use --styleCheck:error for Fluffy + fixes (#2247)
* Use --styleCheck:error for Fluffy + fixes

There seems to be a clash between the names of an object field and
a proc here. As workaround names of the procs are changed.

* Fix case style for rpc client calls in test_portal_testnet

* Fix style case for utp test
2024-05-30 18:12:28 +02:00
Jacek Sieka 919242c98e
results: use canonical import (#2248) 2024-05-30 14:54:03 +02:00
web3-developer 1eb170e686
Fluffy state network tests (#2245)
* Add state network get content test.

* Completed state network get content tests.

* Completed state gossip offer content test.

* Improve state network offer content test.

* Completed state gossip offer test.

* Rename tests.
2024-05-30 17:00:57 +08:00
web3-developer 9354cb8411
Fluffy recursive gossip improvements, fixes and tests. (#2231)
* Refactor get parent gossip code and add Nibbles helper function.

* Add logging to state gossip.

* Unit test recursive gossip using state gossip getParent functions.

* Add recursive gossip genesis json test and fix bug in state gossip getParent.

* Add Nibbles len function.
2024-05-28 08:45:30 +08:00
Kim De Mey 72912626a2
Several spec fixes for HistoricalSummaries + add test-vector test (#2217) 2024-05-24 23:15:04 +02:00
web3-developer 771d6fd3e8
Implementation of Fluffy state network gossip. (#2210)
* Add toContentKey helper functions.

* Implement typed lookup functions for getting account and storage trie nodes and bytecode.

* Re-implementation of recursive gossip.

* Offer validation improvements. Don't allow key end in Nibbles for leafs as defined in the spec.

* Minor refactor to validation and add addition bytecode check.
2024-05-24 15:49:51 +08:00
web3-developer e566e89d03
Fluffy state network refactor (#2200)
* Add additional getTrieProof test.

* Refactor state network: Removed variant objects for values and split state_content into multiple files.
2024-05-22 09:22:07 +08:00
web3-developer 047f8ec4b2
Bump portal spec tests and update state validation tests to use state roots in test vectors. (#2193)
* Bump portal spec tests to version a549242

* Update state validation tests to use state roots in portal spec test vectors.
2024-05-17 23:20:57 +08:00
web3-developer 2891b9aa7d
Fluffy state offer validation (#2170)
* Implemented offer validation interface.

* Implement high level offer validation steps.

* Completed state validation tests.

* Update validation to use result type.

* Update state proof verification tests to test offer verification.

* Query history network to get state root by block hash.

* Fix state network test and remove usage of CoreDb.

* Fix state network gossip test and PR comment updates.

* Add trieproof state validation tests and fix for short nodes.
2024-05-16 16:16:16 +08:00
Kim De Mey 2e01878afe
Add the epoch to the HistoricalSummariesKey (#2179) 2024-05-13 18:49:21 +02:00
web3-developer 5c4358a969
Add retrieval content validation to state network. (#2166)
* Add retrieval content validation to state network.

* Complete state validation tests for retrieval content.

* Improve naming functions and parameters.
2024-05-06 23:13:35 +08:00
web3-developer 7da36bf459
Minor refactor to packNibbles and unpackNibbles and add additional tests. (#2162) 2024-05-03 22:55:51 +08:00
Kim De Mey 9c64b44c89
Add post Bellatrix beacon block proof exports (#2158)
- Add post Bellatrix Beacon block proof exporting to
eth_data_exporter
- Rework the Bellatrix Beacon block proof exports to us yaml
format
- Move some common code of the two different beacon block proofs
to beacon_chain_block_proof_common module
- Move + add to yaml utils to be usable not just for the tests but
also for eth data exporter
2024-05-03 11:50:18 +02:00
Kim De Mey 1512f95067
Remove unneeded nimbus imports (#2154)
Also specifically select from the still required
nimbus imports.
2024-04-25 15:21:09 +02:00
web3-developer d12300ecc9
Remove usages of CoreDb in Fluffy. (#2152)
* Remove references to CoreDb in Fluffy code.

* Add missing import.

* Remove coredb imports and use dynamic server port in tests.

* Fix formatting using nph.
2024-04-25 12:22:21 +02:00
Kim De Mey c610053a09
Prepare Fluffy for usage of historical_roots (#2138) 2024-04-17 17:58:47 +02:00
Kim De Mey 2cbc16be8b
Use test vector files for state content key tests + cleanup (#2119)
- Use yaml test vectors instead of in code data
- Add also test for content id
- Some clean-up + add content key init helper calls
2024-03-29 17:41:19 +01:00
Kim De Mey 8e4368195a
Add an audit mode to portal_bridge history backfill (#2109) 2024-03-26 22:27:31 +01:00
BiKangNing dce00079c4
Remove redundant words (#2100)
* Remove redundant words

Signed-off-by: depthlending <bikangning@outlook.com>

* Fix copyright year

---------

Signed-off-by: depthlending <bikangning@outlook.com>
Co-authored-by: jangko <jangko128@gmail.com>
2024-03-26 07:10:46 +07:00
Kim De Mey 5c1cf0410c
Add exportBlockProofBellatrix command to eth_data_exporter (#2093)
Add functionality to  export bellatrix block proofs from era files
to the eth_data_exporter tool.
2024-03-21 13:25:32 +01:00
Kim De Mey 1159b0114e
Add portal_bridge history mode (#2067)
Portal bridge mode for following latest and injecting the latest
block data into the Portal network.
2024-03-11 18:20:29 +01:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
Kim De Mey b656875778
Quickfix for possible IndexError in state network recursive gossip (#2052)
This would currently manifest sometimes in state gossip test in
CI or locally. But could occur on fluffy binary in case state
network  is enable and data is gossiped.
2024-02-27 17:39:23 +01:00
Daniel Sobol 95a9145f9e
add validators for data received from state network via contentLookup (#2039) 2024-02-20 16:15:24 +03:00
Daniel Sobol 47b254d8b5
State network content retrieval (#2033)
* initial rework of contentLoop for state network

* prepare content for storage and retrieval

* modify test to account for retrieval data
2024-02-19 17:35:53 +03:00
Daniel Sobol 3305c02856
Naive implementation of recursive gossip (#2024) 2024-02-15 18:37:05 +03:00
Daniel Sobol 35131b6d55
nibble encoding update (#2021)
* nibble encoding update

* portal-spec-tests update
2024-02-12 15:39:01 +03:00
Kim De Mey 3f86986658
Fix current fail on excessBlobGas field in block header (#2016) 2024-02-09 15:50:37 +01:00
Kim De Mey 2fb90cfb43
Initial implementation of era1 + export + verify (#1998)
* Initial implementation of era1 + export + verify

Initial implementation of era1 + export and verify command in the
existing eth_data_exporter.

* Quick verify version to test with geth created era1 files

* Add Era1File object and getter for specific block tuple

* Add getAccumulatorRoot and update AccumulatorRoot type name

+ clean-up of some comments.

* Implement BlockIndex based verify + additional helper calls

+ fix bug for reading last (incomplete) Era1 file

* Fix init BlockIndex offsets for last era + extra check on reading

* Remove duplicated e2store code
2024-02-09 11:13:12 +01:00
Daniel Sobol 9c53c73173
state network test vectors update (#2003)
* state network test vectors update
* nibbles codec
* moving test vectors into portal-spec-tests
* fix copyright year in all_fluffy_tests
2024-02-03 00:12:36 +03:00
Kim De Mey a39b51e3e1
Add historical summaries to Portal beacon network (#1990) 2024-01-26 23:38:12 +01:00
Kim De Mey 3ad00686f4
Fix missing ssz encoding overhead in truncateEnrs (#1988) 2024-01-25 12:30:37 +01:00
Kim De Mey 3199857ed5
Further fluffy code clean-up from warnings/hints (#1987) 2024-01-25 11:04:09 +01:00
web3-developer 54644fa3cb
Witness Generation Bug Fixes (#1981)
* Return slots when verifying witness regardless of code length.

* Prevent AccountCache WitnessData codeTouched from being reset.

* Default to using {wfNoFlag} for witness flags in tests to allow running with data from before EIP170.

* Add additional json files to experimental JSON RPC test.

* Use HTTP RPC server in tests.

* Add test to check that block witness contains bytecode.
2024-01-25 01:18:45 +08:00
Kim De Mey dbc1ae86e2
Vendor bumps + related fixes + warning fixes (#1985)
- Vendor bump of stew, nim-eth, chronos, nimbus-eth2 and libp2p
- Bump related fixes + fixes of deprecation warnings
- Several other warnings fixed.
2024-01-24 16:28:03 +01:00
web3-developer 48630ccb03
Experimental RPC endpoints for generating block witnesses (#1977)
* Completed draft implementation of witness JSON-RPC endpoints for portal network bridge.

* Updated Nimbus RPC configuration to support enabling experimental endpoints.

* Moved witness verification tests.

* Added json test for getProof.

* Added main procs to new tests to fix test suite.

* Added getBlockWitness test to blockchain json test suite.

* Added tests for experimental RPC endpoints and improved the API to support returning state proofs from before or after block execution.

* Correctly rollback transaction in getBlockWitness proc.
2024-01-22 17:11:37 +08:00
Daniel Sobol 7ecd38347f
New state network types update (#1976) 2024-01-19 20:18:57 +03:00
Kim De Mey cbf5f59681
Bump chronos and other sub modules + related fixes (#1966)
* Bump chronos and additional modules + fixes

* Additional chronos bump + nim-eth bump for utp and discv5 fixes

* Fix the raw Exception errors in async evm

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-01-12 22:06:19 +01:00
web3-developer a69b1ea90d
Fix clashing import of AccountProof type. (#1962)
* Fix clashing import of AccountProof type.
2024-01-10 16:17:58 +08:00
web3-developer 84e7c680dd
Initial implementation of block witness verification. (#1958)
* Remove support for validation of missing leafs.

* Move proof types into separate file and borrow required procs.

* Added state proof types.

* Completed initial implementation of witness verification.

* Reject empty proofs.

* Added tests for verification of block witnesses.
2024-01-09 23:09:02 +08:00
Daniel Sobol 70013422cd
Initial impl of state network bridge 1902 (#1948) 2024-01-09 12:32:29 +03:00
web3-developer 32e1f94d78
Portal State Proof Verification with Tests (#1951)
* Initial implementation of account and storage proof verification for the portal state network.

* Completed initial state proof verification loop test.

* Completed proof verification tests.

* Minor updates based on PR feedback and comments.

* Add state proof verification test to test runner.
2023-12-21 17:05:53 +08:00
Kim De Mey dded8643d9
Update the beacon and state network content key prefixes (#1950) 2023-12-19 19:59:38 +01:00
Kim De Mey 2b71e3e600
Remove contentSize and contentCount from history network log (#1933) 2023-12-11 22:00:56 +01:00
Kim De Mey 78ec80eea0
Add trace logs when ENR gets added to portal network routing table (#1932) 2023-12-11 11:53:32 +01:00
Kim De Mey aad4adde07
Move all Fluffy database code to database folder (#1918) 2023-12-01 17:20:52 +01:00
Kim De Mey d4fc234e53
Fix missing labels for Portal wire protocol metrics (#1916) 2023-12-01 14:59:17 +01:00
Daniel Sobol 47e03919fe
always return tracing result even if no content was found (#1908) 2023-11-27 10:21:19 +01:00
Kim De Mey 6d1622875a
Do a radius check before accessing the db in handleFindContent (#1906) 2023-11-24 15:07:23 +01:00
Kim De Mey cf9a44f78e
Portal protocol clean-up for some routing table calls (#1904) 2023-11-23 22:20:23 +01:00
Kim De Mey 11e3171723
Add metrics to track the distance of content requested and offered (#1903) 2023-11-23 18:49:15 +01:00
Kim De Mey a7bb52e5b5
Add force prune for statically set radius (#1896) 2023-11-21 16:16:15 +01:00
Kim De Mey 6338969dd5
Lower the content deletion fraction and other contentdb clean-up (#1893)
- The current setting of 0.25 was very big. Set to 0.5, which is
potentially still large. This change did expose some issues with
the current implementation and especially testing.
- General clean-up, renaming for consistency, and
re-ordering/restructuring/deletion of some code.
- Fixed several typos
- ...
2023-11-16 15:27:30 +01:00
Kim De Mey 03a739ff1b
Add contentdb contentCount and contentdb renames/cleanup (#1886) 2023-11-07 19:46:26 +01:00
Daniel Sobol 9022a3993c
implement portal_historyTraceRecursiveFindContent (#1813)
implement portal_historyTraceRecursiveFindContent according to spec
2023-10-30 17:48:06 +03:00
Kim De Mey 9563beea56
Don't send the processed bitlist if contentQueue is full (#1835)
This would lead to sending an AcceptMessage with a zeroed out
connection id but still potentially bits set in the bitlist.
Requesting nodes could try to open up an uTP connection that
would never be accepted on the responder side.
2023-10-20 18:03:20 +02:00
Kim De Mey b7d0b06e79
Change some Fluffy related cli defaults (#1833) 2023-10-20 14:30:21 +02:00
Kim De Mey 436ef0e40e
Portal BeaconLightClientNetwork -> BeaconNetwork renaming (#1832) 2023-10-20 12:06:25 +02:00
Kim De Mey 14ee5f6820
Clean-up/refactor in beacon lc db (#1830) 2023-10-19 15:47:29 +02:00
Kim De Mey 6199ddc1c9
Better validation logs for Portal beacon network (#1823) 2023-10-18 23:29:20 +02:00
Kim De Mey 0472b75e23
Rework fluffy beacon lc db to allow storing processed content (#1821) 2023-10-18 16:59:44 +02:00
andri lim 04c7ed8ec9
Change EthTime from std.Time to distinct uint64 (#1820)
* Change EthTime from std.Time to distinct uint64

* Bump nimbus-eth2
2023-10-18 09:16:11 +07:00
Kim De Mey ca61a7009d
Bump submodules and related fixes (#1729)
* Bump submodules and related fixes

* Fix some warnings due to chronos and Nim bumps

* Seems macOS and Windows have different behavior
2023-10-17 14:19:50 +02:00
Kim De Mey effe9c7ad8
Implement safe and finalized tag for eth_getBlockByNumber (#1814)
Only available when running the beacon LC and the portal beacon
network.
2023-10-09 17:49:15 +02:00
Kim De Mey 7b313ec1ac
Use signature slot instead of attested header slot (#1811)
Use signature slot instead of attested header slot in the
OptimisticUpdate content key.
2023-10-06 17:42:34 +02:00
Kim De Mey f72f02c88b
Remove optimistic slot from FinalityUpdate content key (#1810) 2023-10-06 15:46:53 +02:00
Kim De Mey 43b6014ffe
Further Portal beacon network improvements (#1808) 2023-10-05 19:29:39 +02:00
Kim De Mey f433297631
Improvements to portal beacon network (#1806)
- Adjust beacon lc db to avoid requirement of pruning optimistic
and finality updates
- Enable Portal config for beacon chain network to apply IP limits
configs on routing table
2023-10-05 08:57:45 +02:00
Daniel Sobol 20885f4ca5
don't ping nodes during revalidation if they were pinged recently (#1779) 2023-10-03 19:48:00 +03:00
Kim De Mey 2f2c5127ea
Further implementation to working prototype for Portal beacon LC (#1781)
- Link network gossip validation to LC processor validation
- QuickFix put/get for optimistic and finality updates
- Minor fixes and clean-up
- Improve bridge gossip for new LC Updates
- Adjust local testnet script to be able to locally test this
- Adjust test and skip broken test
2023-09-28 18:16:41 +02:00
Kim De Mey c405f79a84
Update the portal beacon lc manager (#1770)
Similar changes sd done on the libp2p version. To be able to have
same improvements and reuse more of the lc sync helpers code.
2023-09-22 18:42:02 +02:00
andri lim 7af9e3dc53
Refactor beacon skeleton (#1761)
* Fix unlisted exception due to recent modification in CoreDB interface

* Refactor beacon skeleton

* More unlisted exception fix
2023-09-19 11:52:28 +07:00
Kim De Mey 18b3ff3390
Correct the shortcut for ommershash verification post Shanghai (#1756)
* Correct the shortcut for ommershash verification post Shanghai

There was a faulty length check being done in the assumption that
the uncles field was already the list. However it is still the
RLP encoded data at that point, hence we check directly for the
RLP encoded value for empty list.

* Fix copy-paste error in withdrawal root verification

And print out the hashes in txs root and withdrawal root
verification
2023-09-18 14:50:07 +02:00
Kim De Mey a65b1c9062
Do not accept new offers if our contentQueue is full (#1753) 2023-09-16 16:06:11 +02:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
Kim De Mey c04f2477ac
Add version to fluffy cli flags and metrics (#1736)
Also update Grafana dashboard to incorperate the metrics and do
a confutils module bump.
2023-09-07 16:27:13 +02:00
Daniel Sobol 69a1a988b0
don't gossip data back to the node we rcvd it from [#1227] (#1719) 2023-09-04 12:21:01 +02:00
Kim De Mey 7ae333ac8b
Add discv5 provided ENR directly to Portal protocol routing table (#1714)
* Add discv5 provided ENR directly to Portal protocol routing table

Previous version of getting the ENR from the discv5 routing table
would not work due to the order of first calling the talk protocol
handler and only after that the addEnr to the disc5 routing table.

Instead of changing this order, pass along the ENR directly to
avoid this additional getNode call.

* Still request ENR from discv5 if it wasn't passed via handshake
2023-08-30 20:38:15 +02:00
Daniel Sobol 85134eb24b
add hidden --disable-poke option into fluffy [#1640] (#1711) 2023-08-30 10:01:00 +02:00
Kim De Mey 7aead6151e
Add utpTransfer bool to recursiveFindContent JSON-RPC result (#1710)
Rename portal protocol ContentInfo to ContentKV in the
process, due to duplicate types and simply better name
for this object.
2023-08-24 18:19:29 +02:00
Kim De Mey 6535b39e47
Add SszError to message decoding result (#1660)
This allows for improved error messages. Required changing the PortalResult from cstring to string for the error.
2023-08-23 18:03:09 +02:00
andri lim bdaeedb09f
rename data gas to blob gas (#1659)
* rename data gas to blob gas

* bump more submodules
* extend evmc tx_context with EIP-4844 blob_hashes
2023-08-04 19:43:30 +07:00
Jordan Hrycaj 221e6c9e2f
Unified database frontend integration (#1670)
* Nimbus folder environment update

details:
* Integrated `CoreDbRef` for the sources in the `nimbus` sub-folder.
* The `nimbus` program does not compile yet as it needs the updates
  in the parallel `stateless` sub-folder.

* Stateless environment update

details:
* Integrated `CoreDbRef` for the sources in the `stateless` sub-folder.
* The `nimbus` program compiles now.

* Premix environment update

details:
* Integrated `CoreDbRef` for the sources in the `premix` sub-folder.

* Fluffy environment update

details:
* Integrated `CoreDbRef` for the sources in the `fluffy` sub-folder.

* Tools environment update

details:
* Integrated `CoreDbRef` for the sources in the `tools` sub-folder.

* Nodocker environment update

details:
* Integrated `CoreDbRef` for the sources in the
  `hive_integration/nodocker` sub-folder.

* Tests environment update

details:
* Integrated `CoreDbRef` for the sources in the `tests` sub-folder.
* The unit tests compile and run cleanly now.

* Generalise `CoreDbRef` to any `select_backend` supported database

why:
  Generalisation was just missed due to overcoming some compiler oddity
  which was tied to rocksdb for testing.

* Suppress compiler warning for `newChainDB()`

why:
  Warning was added to this function which must be wrapped so that
  any `CatchableError` is re-raised as `Defect`.

* Split off persistent `CoreDbRef` constructor into separate file

why:
  This allows to compile a memory only database version without linking
  the backend library.

* Use memory `CoreDbRef` database by default

detail:
 Persistent DB constructor needs to import `db/core_db/persistent

why:
 Most tests use memory DB anyway. This avoids linking `-lrocksdb` or
 any other backend by default.

* fix `toLegacyBackend()` availability check

why:
  got garbled after memory/persistent split.

* Clarify raw access to MPT for snap sync handler

why:
  Logically, `kvt` is not the raw access for the hexary trie (although
  this holds for the legacy database)
2023-08-04 12:10:09 +01:00
Kim De Mey 89b3e679e2
Add new portal beacon chain lc content encoding tests (#1636) 2023-07-08 17:01:33 +02:00
Kim De Mey 8ebac4c878
Some more Options to Opt and similar changes (#1611)
Includes nim-eth bump.
2023-06-27 19:43:32 +02:00
Kim De Mey 2de9c95770
Remove a socket.close after destroy + bump nim-eth for uTP fixes (#1604) 2023-06-21 18:56:57 +02:00
Kim De Mey dd4da7491d
Remove yield usage from portal stream and protocol (#1602) 2023-06-15 14:32:33 +02:00
jangko 67aaf92c1d
bump submodules 2023-06-07 18:12:02 +07:00
Kim De Mey bf21484f70
Add EL block proofs verifable from historical_summaries (#1588) 2023-05-30 07:02:39 +02:00
Kim De Mey c9f3f82877
Add new Portal BlockBody type for Shanghai fork (#1589) 2023-05-30 06:56:54 +02:00
Kim De Mey 33fd8b1fae
Add historical_roots with proof + test (#1579) 2023-05-11 19:08:34 +02:00
Kim De Mey 408394a2bd
Bump nim-eth and remove unneeded Defect raises (#1575) 2023-05-10 18:04:35 +02:00
Kim De Mey 2ba974a7cd
Adjust portal FindContent JSON-RPC according to latest specs (#1566)
Also remove the FindContentFull unspecced JSON-RPC as the new
FindContent specification replaces this functionality basically.
2023-04-28 09:04:21 +02:00
Kim De Mey 3eafb15d7c
Make beacon LC adjustements to support capella in light clients (#1545)
- Bump nimbus-eth2 repo
- Beacon LC adjustements in fluffy
- Beacon LC adjustements in nimbus_verified_proxy
2023-04-13 18:40:01 +02:00
Kim De Mey 52d8308821
Some quick fixes for Shanghai support in Fluffy (#1541)
- Remove failing on withdrawalsRoot to allow Shanghai BlockHeader.
Still need to add real checks in block header / body validation.
- Add withdrawals array to Block object for JSON-RPC API
2023-04-12 14:20:52 +02:00
Kim De Mey 034d0e6305
Add exporting of headersWithProof and configurable web3 url (#1539) 2023-04-11 14:06:45 +02:00
Kim De Mey fdad9cfd00
Move content offered log after socket close wait (#1530)
To be clear that the socket also got closed before this.
Also explain that on socket close, the actual socket still waits
for an ACK.
2023-04-04 17:36:57 +02:00
Kim De Mey 9453d5bb3c
Add loop to history network to log status message on interval (#1526) 2023-03-31 18:52:21 +02:00