Commit Graph

438 Commits

Author SHA1 Message Date
Kim De Mey df2dcf593c
Remove few more unneeded nimbus imports in Fluffy tests (#2159) 2024-05-03 12:31:54 +02:00
Kim De Mey 9c64b44c89
Add post Bellatrix beacon block proof exports (#2158)
- Add post Bellatrix Beacon block proof exporting to
eth_data_exporter
- Rework the Bellatrix Beacon block proof exports to us yaml
format
- Move some common code of the two different beacon block proofs
to beacon_chain_block_proof_common module
- Move + add to yaml utils to be usable not just for the tests but
also for eth data exporter
2024-05-03 11:50:18 +02:00
Kim De Mey 1512f95067
Remove unneeded nimbus imports (#2154)
Also specifically select from the still required
nimbus imports.
2024-04-25 15:21:09 +02:00
web3-developer d12300ecc9
Remove usages of CoreDb in Fluffy. (#2152)
* Remove references to CoreDb in Fluffy code.

* Add missing import.

* Remove coredb imports and use dynamic server port in tests.

* Fix formatting using nph.
2024-04-25 12:22:21 +02:00
Kim De Mey 1d90b924ee
Adjust color scheme for fluffy.guide (#2144)
Use the same scheme as for the nimbus.guide.
Two differences however:
- Set the dark scheme as default, considering the new nimbus.team
website also has dark as default.
- Remove the orange/amber for the light scheme, considering the
light version of nimbus.team has no more orange.
2024-04-22 15:33:51 +02:00
Jordan Hrycaj e8eb3268f5
Generalise prune mode option 4 different db models (#2139)
* Update README

* Nimbus-main: replaced `PruneMode` options by `ChainDbMode` options

details:
  For the legacy database, this changes the phrase
  - `conf.pruneMode == PruneMode.Full` to the expression
  + `conf.chainDbMode == ChainDbMode.Prune`.

* Fix issues moaned about by NIM compiler

* Fix copyright year
2024-04-17 18:09:55 +00:00
Kim De Mey c610053a09
Prepare Fluffy for usage of historical_roots (#2138) 2024-04-17 17:58:47 +02:00
Kim De Mey d0ac3a1986
Add docs on pruning in Fluffy (#2137) 2024-04-17 14:06:53 +02:00
Kim De Mey 6c2c99a9ae
Add reclaim to the fcli_db prune cmd (#2122) 2024-04-02 20:50:05 +02:00
Kim De Mey 2cbc16be8b
Use test vector files for state content key tests + cleanup (#2119)
- Use yaml test vectors instead of in code data
- Add also test for content id
- Some clean-up + add content key init helper calls
2024-03-29 17:41:19 +01:00
Kim De Mey 095bcae8b7
Use yaml versions of the portal beacon lc test vectors (#2117) 2024-03-29 17:12:23 +01:00
Kim De Mey 058f412bbc
Small adjustment to portal_bridge + docs update (#2110) 2024-03-27 19:16:24 +01:00
Kim De Mey 8e4368195a
Add an audit mode to portal_bridge history backfill (#2109) 2024-03-26 22:27:31 +01:00
BiKangNing dce00079c4
Remove redundant words (#2100)
* Remove redundant words

Signed-off-by: depthlending <bikangning@outlook.com>

* Fix copyright year

---------

Signed-off-by: depthlending <bikangning@outlook.com>
Co-authored-by: jangko <jangko128@gmail.com>
2024-03-26 07:10:46 +07:00
Kim De Mey 1c98733d4d
Move portal bridge beacon code and add skeleton for state bridge (#2101) 2024-03-22 19:01:22 +01:00
Kim De Mey 1379e05db1
Add bellatrix block proof test using new test vectors (#2099) 2024-03-22 13:05:06 +01:00
Kim De Mey d3a706c229
Replace status-im/portal-spec-tests with ethereum fork version (#2097)
- The fluffy test vector repo got forked (well, copied rather) to
become the official one under ethereum github org, so we change
to that repo now and archive ours.
- Our repo also stored accumulator / historical_roots, replace
that with a new repo which is only for network configs.
- Several changes needed to be made due to test vectors that got
updated + some of them got changed to / are yaml format instead of
json.
2024-03-22 11:28:44 +01:00
Kim De Mey 5c1cf0410c
Add exportBlockProofBellatrix command to eth_data_exporter (#2093)
Add functionality to  export bellatrix block proofs from era files
to the eth_data_exporter tool.
2024-03-21 13:25:32 +01:00
andri lim 30277be1f3
Bump nim-web3 to 285d97c2b05bbe2a13dab4b52ea878157fb1a1a1 (#2088)
* Bump nim-web3 to 285d97c2b05bbe2a13dab4b52ea878157fb1a1a1

Unify EthCall/EthSend into TransactionArgs (#138)

* bump ssz-serialization

* Fix BlockNumber conversion

* Bump ssz-serialization: Restrict toSszType usage to non SszType in readSszBytes (#81)
2024-03-21 08:05:22 +07:00
Kim De Mey 49a199dbc3
Add beacon state historical_roots to fluffy (#2085)
- Add historical_roots in the binary for proof verification
pre-capella
- Add command to export historical_roots to file in
eth_data_exporter tool
2024-03-19 16:45:32 +01:00
Kim De Mey bfa756c36b
Bump json-rpc and web3 and remove unneeded errors import/export (#2078) 2024-03-15 11:44:18 +01:00
Kim De Mey 461c7e7d85
Add Era1 backfill to portal_bridge history mode (#2077) 2024-03-15 10:41:41 +01:00
Kim De Mey 0fb7632aaa
Fix BlockNumber conversion for getBlockReceipts in portal_bridge (#2074) 2024-03-14 17:21:10 +01:00
Kim De Mey 7287efc7b4
Clean-up of some Portal json-rpc code (#2073)
- Use the new createRpcSigsFromNim for client json-rpc API
- Avoid importing any nimbus/rpc specifics, use only web3 and
fluffy local rpc code
- Adjust tools making use of the client side API
2024-03-13 16:58:50 +01:00
Kim De Mey 849607cfd1
Remove option of using beacon_lc_bridge together with a full node (#2070)
Keep the beacon_lc_bridge a fully standalone bridge, and instead
use the new portal_bridge which works by means of full nodes or
web3 providers.
2024-03-12 14:06:22 +01:00
Kim De Mey 1159b0114e
Add portal_bridge history mode (#2067)
Portal bridge mode for following latest and injecting the latest
block data into the Portal network.
2024-03-11 18:20:29 +01:00
Kim De Mey f8fca07d37
Update protocol-interop-testing.md with correct dcli build dir (#2064) 2024-03-08 10:32:53 +01:00
Kim De Mey 88a93beb26
Provide a specific error in JSON-RPC API when content is not found (#2059) 2024-03-06 08:57:09 +01:00
Kim De Mey 0debf1a122
Introduce --networks and --portal-network cli options (#2061)
The --networks option is to select the networks alias the Portal
sub-protocols.
The --network option is deprecated and replaced with the
--portal-network option to avoid confusion with --networks option.

This commit also remove the --state flag which was a temporary
quick-fix.
2024-03-05 13:45:48 +01:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
Kim De Mey b656875778
Quickfix for possible IndexError in state network recursive gossip (#2052)
This would currently manifest sometimes in state gossip test in
CI or locally. But could occur on fluffy binary in case state
network  is enable and data is gossiped.
2024-02-27 17:39:23 +01:00
Daniel Sobol 95a9145f9e
add validators for data received from state network via contentLookup (#2039) 2024-02-20 16:15:24 +03:00
Daniel Sobol 47b254d8b5
State network content retrieval (#2033)
* initial rework of contentLoop for state network

* prepare content for storage and retrieval

* modify test to account for retrieval data
2024-02-19 17:35:53 +03:00
Daniel Sobol 2766e20179
Add flag to explicitly enable state network (#2031) 2024-02-17 01:38:58 +03:00
Kim De Mey 9378774b9f
Add Era1 based gossip calls + Era1 helpers (#2029)
- Add Era1 helpers to be able to iterate fast over block tuples
and individual total difficulties
- Add buildAccumulator from Era1 file
- Add Era1 based BlockHeader with proof + bodies/receipts gossip
calls
- Add new JSON-RPC debug methods to be able to test the above
with a standalone fluffy node
2024-02-15 16:49:22 +01:00
Daniel Sobol 3305c02856
Naive implementation of recursive gossip (#2024) 2024-02-15 18:37:05 +03:00
Daniel Sobol 35131b6d55
nibble encoding update (#2021)
* nibble encoding update

* portal-spec-tests update
2024-02-12 15:39:01 +03:00
Kim De Mey 3f86986658
Fix current fail on excessBlobGas field in block header (#2016) 2024-02-09 15:50:37 +01:00
Kim De Mey 2fb90cfb43
Initial implementation of era1 + export + verify (#1998)
* Initial implementation of era1 + export + verify

Initial implementation of era1 + export and verify command in the
existing eth_data_exporter.

* Quick verify version to test with geth created era1 files

* Add Era1File object and getter for specific block tuple

* Add getAccumulatorRoot and update AccumulatorRoot type name

+ clean-up of some comments.

* Implement BlockIndex based verify + additional helper calls

+ fix bug for reading last (incomplete) Era1 file

* Fix init BlockIndex offsets for last era + extra check on reading

* Remove duplicated e2store code
2024-02-09 11:13:12 +01:00
Daniel Sobol 9c53c73173
state network test vectors update (#2003)
* state network test vectors update
* nibbles codec
* moving test vectors into portal-spec-tests
* fix copyright year in all_fluffy_tests
2024-02-03 00:12:36 +03:00
Kim De Mey a39b51e3e1
Add historical summaries to Portal beacon network (#1990) 2024-01-26 23:38:12 +01:00
Kim De Mey 3ad00686f4
Fix missing ssz encoding overhead in truncateEnrs (#1988) 2024-01-25 12:30:37 +01:00
Kim De Mey 3199857ed5
Further fluffy code clean-up from warnings/hints (#1987) 2024-01-25 11:04:09 +01:00
web3-developer 54644fa3cb
Witness Generation Bug Fixes (#1981)
* Return slots when verifying witness regardless of code length.

* Prevent AccountCache WitnessData codeTouched from being reset.

* Default to using {wfNoFlag} for witness flags in tests to allow running with data from before EIP170.

* Add additional json files to experimental JSON RPC test.

* Use HTTP RPC server in tests.

* Add test to check that block witness contains bytecode.
2024-01-25 01:18:45 +08:00
Kim De Mey dbc1ae86e2
Vendor bumps + related fixes + warning fixes (#1985)
- Vendor bump of stew, nim-eth, chronos, nimbus-eth2 and libp2p
- Bump related fixes + fixes of deprecation warnings
- Several other warnings fixed.
2024-01-24 16:28:03 +01:00
web3-developer 48630ccb03
Experimental RPC endpoints for generating block witnesses (#1977)
* Completed draft implementation of witness JSON-RPC endpoints for portal network bridge.

* Updated Nimbus RPC configuration to support enabling experimental endpoints.

* Moved witness verification tests.

* Added json test for getProof.

* Added main procs to new tests to fix test suite.

* Added getBlockWitness test to blockchain json test suite.

* Added tests for experimental RPC endpoints and improved the API to support returning state proofs from before or after block execution.

* Correctly rollback transaction in getBlockWitness proc.
2024-01-22 17:11:37 +08:00
Daniel Sobol 7ecd38347f
New state network types update (#1976) 2024-01-19 20:18:57 +03:00
Kim De Mey 2df9e2b7b1
Cleanup utp_test and make the code more flexible to change amounts (#1972) 2024-01-15 17:29:29 +01:00
jangko 3e21281d12
Bump nim-web3 and others
Bump nim-json-rpc and nimbus-eth2 too.
Reason: both nim-json-rpc and nim-web3 migrate from
stdlib/json to nim-json-serialization
2024-01-14 10:41:23 +07:00
Kim De Mey cbf5f59681
Bump chronos and other sub modules + related fixes (#1966)
* Bump chronos and additional modules + fixes

* Additional chronos bump + nim-eth bump for utp and discv5 fixes

* Fix the raw Exception errors in async evm

---------

Co-authored-by: jangko <jangko128@gmail.com>
2024-01-12 22:06:19 +01:00
web3-developer a69b1ea90d
Fix clashing import of AccountProof type. (#1962)
* Fix clashing import of AccountProof type.
2024-01-10 16:17:58 +08:00
web3-developer 84e7c680dd
Initial implementation of block witness verification. (#1958)
* Remove support for validation of missing leafs.

* Move proof types into separate file and borrow required procs.

* Added state proof types.

* Completed initial implementation of witness verification.

* Reject empty proofs.

* Added tests for verification of block witnesses.
2024-01-09 23:09:02 +08:00
Daniel Sobol 70013422cd
Initial impl of state network bridge 1902 (#1948) 2024-01-09 12:32:29 +03:00
jangko c0d52ba179
Get rid of HexQuantityStr and HexDataStr usage
Both types are not safe and require validation/conversion
from rpc implementer.
This PR change it to safer types and delegate the conversion
and validation to the rpc library.
2023-12-27 07:48:53 +07:00
web3-developer 32e1f94d78
Portal State Proof Verification with Tests (#1951)
* Initial implementation of account and storage proof verification for the portal state network.

* Completed initial state proof verification loop test.

* Completed proof verification tests.

* Minor updates based on PR feedback and comments.

* Add state proof verification test to test runner.
2023-12-21 17:05:53 +08:00
Kim De Mey dded8643d9
Update the beacon and state network content key prefixes (#1950) 2023-12-19 19:59:38 +01:00
Kim De Mey 3675cd6736
Update protocol-interop-testing.md with fixed dcli build location (#1946) 2023-12-15 09:22:03 +01:00
Daniel Sobol 5b448e87aa
trin's bootstrap ENR update. Fix #1926 (#1927) 2023-12-15 09:04:26 +01:00
Kim De Mey 64bfe4d63a
Fix missing parameter passing for fullTransactions in JSON-RPC API (#1942) 2023-12-13 14:49:10 +01:00
jangko 5fd54961e3
Bump nim-web3: remove rpc types duplicates 2023-12-13 07:58:12 +07:00
Kim De Mey c99c0687f8
Rename beacon bridge to portal_bridge and prep for 1 bridge binary (#1936)
Renaming the beacon_chain_bridge to portal_bridge and preparing
for stuffing all bridge functionality for all networks under this
2023-12-12 21:08:58 +01:00
Kim De Mey 2b71e3e600
Remove contentSize and contentCount from history network log (#1933) 2023-12-11 22:00:56 +01:00
Daniel Sobol dc55ee0c87
fix local dev container for portal-hive (#1931)
* fix local dev container for portal-hive
2023-12-11 14:56:00 +01:00
Kim De Mey 78ec80eea0
Add trace logs when ENR gets added to portal network routing table (#1932) 2023-12-11 11:53:32 +01:00
Kim De Mey b27c2ff449
Add fcli_db tool for offline interactions with a fluffy ContentDb (#1929) 2023-12-07 13:08:49 +01:00
Kim De Mey 443e7c49a9
Fix state network RPC methods in interop docs (#1922) 2023-12-04 16:27:04 +01:00
Kim De Mey aad4adde07
Move all Fluffy database code to database folder (#1918) 2023-12-01 17:20:52 +01:00
Kim De Mey d4fc234e53
Fix missing labels for Portal wire protocol metrics (#1916) 2023-12-01 14:59:17 +01:00
Kim De Mey 49d6c5728b
Bump nim-ssz-ser for fix for Empty SSZ list in Union + add test (#1909) 2023-12-01 13:04:34 +01:00
Daniel Sobol 47e03919fe
always return tracing result even if no content was found (#1908) 2023-11-27 10:21:19 +01:00
Kim De Mey 6d1622875a
Do a radius check before accessing the db in handleFindContent (#1906) 2023-11-24 15:07:23 +01:00
Kim De Mey cf9a44f78e
Portal protocol clean-up for some routing table calls (#1904) 2023-11-23 22:20:23 +01:00
Kim De Mey 11e3171723
Add metrics to track the distance of content requested and offered (#1903) 2023-11-23 18:49:15 +01:00
Kim De Mey c46706ac75
Add force pruning for the storage-capacity option (#1901) 2023-11-23 11:44:57 +01:00
Kim De Mey a7bb52e5b5
Add force prune for statically set radius (#1896) 2023-11-21 16:16:15 +01:00
Kim De Mey 6338969dd5
Lower the content deletion fraction and other contentdb clean-up (#1893)
- The current setting of 0.25 was very big. Set to 0.5, which is
potentially still large. This change did expose some issues with
the current implementation and especially testing.
- General clean-up, renaming for consistency, and
re-ordering/restructuring/deletion of some code.
- Fixed several typos
- ...
2023-11-16 15:27:30 +01:00
Kim De Mey 86f93fdba5
ValidIpAddress -> IpAddress related vendor bumps and changes (#1892) 2023-11-10 19:38:11 +01:00
Kim De Mey c41d531c51
Adjust storage capacity option (#1891)
- Rename storage-size to storage capacity
- Change type to uint64 to avoid potential wrap around
- Change to MB instead of KB and set default to 2GB
2023-11-10 17:16:15 +01:00
Kim De Mey 03a739ff1b
Add contentdb contentCount and contentdb renames/cleanup (#1886) 2023-11-07 19:46:26 +01:00
Daniel Sobol 9022a3993c
implement portal_historyTraceRecursiveFindContent (#1813)
implement portal_historyTraceRecursiveFindContent according to spec
2023-10-30 17:48:06 +03:00
Jakub Sokołowski 2a76975f6a
add CNAME file to define custom domain for GH Pages
https://www.mkdocs.org/user-guide/deploying-your-docs/#custom-domains

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-10-25 14:28:42 +02:00
Kim De Mey 964141d5ba
Fix broken link in book, take 2, oops (#1860) 2023-10-25 13:28:52 +02:00
Kim De Mey 75b62b45f1
Dead link fixes and other small improvements to the fluffy book (#1859) 2023-10-25 13:23:17 +02:00
Kim De Mey 6e59cccc5b
Add more documentation to the fluffy book and reorganize it (#1856)
Moved pretty much everything from the readme.md file now to the
book and structured it better to be more accessible.
2023-10-24 18:18:30 +02:00
Kim De Mey 5048c87679
First go at the Fluffy book using mkdocs as for nimbus.guide (#1842)
* First go at the Fluffy book using mkdocs as for nimbus.guide

* reorganize a little the index page

* Add small paragraph on the what in index page
2023-10-23 15:29:54 +02:00
Kim De Mey 9563beea56
Don't send the processed bitlist if contentQueue is full (#1835)
This would lead to sending an AcceptMessage with a zeroed out
connection id but still potentially bits set in the bitlist.
Requesting nodes could try to open up an uTP connection that
would never be accepted on the responder side.
2023-10-20 18:03:20 +02:00
Kim De Mey b7d0b06e79
Change some Fluffy related cli defaults (#1833) 2023-10-20 14:30:21 +02:00
Kim De Mey 436ef0e40e
Portal BeaconLightClientNetwork -> BeaconNetwork renaming (#1832) 2023-10-20 12:06:25 +02:00
Kim De Mey 14ee5f6820
Clean-up/refactor in beacon lc db (#1830) 2023-10-19 15:47:29 +02:00
Kim De Mey 6199ddc1c9
Better validation logs for Portal beacon network (#1823) 2023-10-18 23:29:20 +02:00
Kim De Mey 0472b75e23
Rework fluffy beacon lc db to allow storing processed content (#1821) 2023-10-18 16:59:44 +02:00
andri lim 04c7ed8ec9
Change EthTime from std.Time to distinct uint64 (#1820)
* Change EthTime from std.Time to distinct uint64

* Bump nimbus-eth2
2023-10-18 09:16:11 +07:00
Kim De Mey ca61a7009d
Bump submodules and related fixes (#1729)
* Bump submodules and related fixes

* Fix some warnings due to chronos and Nim bumps

* Seems macOS and Windows have different behavior
2023-10-17 14:19:50 +02:00
Kim De Mey 3444ffaf30
Update test to latest beacon LC test vectors (#1815) 2023-10-11 15:44:23 +02:00
Kim De Mey effe9c7ad8
Implement safe and finalized tag for eth_getBlockByNumber (#1814)
Only available when running the beacon LC and the portal beacon
network.
2023-10-09 17:49:15 +02:00
Kim De Mey 7b313ec1ac
Use signature slot instead of attested header slot (#1811)
Use signature slot instead of attested header slot in the
OptimisticUpdate content key.
2023-10-06 17:42:34 +02:00
Kim De Mey f72f02c88b
Remove optimistic slot from FinalityUpdate content key (#1810) 2023-10-06 15:46:53 +02:00
Kim De Mey 43b6014ffe
Further Portal beacon network improvements (#1808) 2023-10-05 19:29:39 +02:00
Kim De Mey f433297631
Improvements to portal beacon network (#1806)
- Adjust beacon lc db to avoid requirement of pruning optimistic
and finality updates
- Enable Portal config for beacon chain network to apply IP limits
configs on routing table
2023-10-05 08:57:45 +02:00
Daniel Sobol 20885f4ca5
don't ping nodes during revalidation if they were pinged recently (#1779) 2023-10-03 19:48:00 +03:00