Commit Graph

52 Commits

Author SHA1 Message Date
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Kim De Mey d814d84b9b
Use --styleCheck:error for Fluffy + fixes (#2247)
* Use --styleCheck:error for Fluffy + fixes

There seems to be a clash between the names of an object field and
a proc here. As workaround names of the procs are changed.

* Fix case style for rpc client calls in test_portal_testnet

* Fix style case for utp test
2024-05-30 18:12:28 +02:00
Kim De Mey 7287efc7b4
Clean-up of some Portal json-rpc code (#2073)
- Use the new createRpcSigsFromNim for client json-rpc API
- Avoid importing any nimbus/rpc specifics, use only web3 and
fluffy local rpc code
- Adjust tools making use of the client side API
2024-03-13 16:58:50 +01:00
Kim De Mey 0debf1a122
Introduce --networks and --portal-network cli options (#2061)
The --networks option is to select the networks alias the Portal
sub-protocols.
The --network option is deprecated and replaced with the
--portal-network option to avoid confusion with --networks option.

This commit also remove the --state flag which was a temporary
quick-fix.
2024-03-05 13:45:48 +01:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
Daniel Sobol 2766e20179
Add flag to explicitly enable state network (#2031) 2024-02-17 01:38:58 +03:00
Kim De Mey 64bfe4d63a
Fix missing parameter passing for fullTransactions in JSON-RPC API (#1942) 2023-12-13 14:49:10 +01:00
jangko 5fd54961e3
Bump nim-web3: remove rpc types duplicates 2023-12-13 07:58:12 +07:00
Kim De Mey c99c0687f8
Rename beacon bridge to portal_bridge and prep for 1 bridge binary (#1936)
Renaming the beacon_chain_bridge to portal_bridge and preparing
for stuffing all bridge functionality for all networks under this
2023-12-12 21:08:58 +01:00
Kim De Mey aad4adde07
Move all Fluffy database code to database folder (#1918) 2023-12-01 17:20:52 +01:00
Kim De Mey b7d0b06e79
Change some Fluffy related cli defaults (#1833) 2023-10-20 14:30:21 +02:00
Kim De Mey 2f2c5127ea
Further implementation to working prototype for Portal beacon LC (#1781)
- Link network gossip validation to LC processor validation
- QuickFix put/get for optimistic and finality updates
- Minor fixes and clean-up
- Improve bridge gossip for new LC Updates
- Adjust local testnet script to be able to locally test this
- Adjust test and skip broken test
2023-09-28 18:16:41 +02:00
Kim De Mey 31cff910b1
Rename beacon_chain_bridge to beacon_lc_bridge (#1655) 2023-07-25 20:53:31 +02:00
Kim De Mey c20892f805
Allow starting beacon network in local testnet and add lc bridge (#1654)
Add the option to start the fluffy nodes with the beacon network
activated and allow for running a beacon lc bridge in the local
testnet script.
2023-07-25 14:52:44 +02:00
Kim De Mey edc87b702b
Launch Fluffy builds directly from make to avoid compile issue (#1646)
* Launch Fluffy builds directly from make to avoid compile issue

Without this change, builds on latest macos fails when ulimit is
not set to 1024. But it will still cause libbacktrace error to occur
when launching the binaries so it would be still advised to
set it to 1024.

* Fix fluffy local testnet for some macOS systems

And some additional improvements to the script + run the fluffy
nodes at INFO log-level to speed-up the testing time.

* Split up fluffy tests in separate targets

This way the two test binaries can be build and ran
concurrently.
2023-07-21 16:30:22 +02:00
Kim De Mey f950e0aac8
Remove unused variable in local testnet script (#1641) 2023-07-13 13:43:50 +02:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
Kim De Mey 12f66ae598
Move BlockHeaderWithProof content to content key selector 0 (#1379)
* Move BlockHeaderWithProof content to content key selector 0

- Remove as content type with content key selector 4
- Replace regular block header with BlockHeaderWithProof at
content key selector 0

* Apply blockHeader content key also to bridge

* Add tests for header with proof generation and verification
2023-01-20 23:04:58 +01:00
Kim De Mey 52a2257922
Adjust Portal JSON-RPC Offer call and add Gossip call (#1418) 2023-01-17 14:47:10 +01:00
Kim De Mey 90f91e5ebd
Reorganize eth data files parsing code (#1407) 2023-01-05 15:26:58 +01:00
Kim De Mey c962bafd5a
Fix Portal Hive fails by correcting Portal history JSON RPC API (#1365)
* Fix Portal Hive fails by correcting Portal history JSON RPC API

- Field naming in discv5_nodeInfo
- Call naming of portal_historyStore
- Other: some proc to func adjustements
2022-12-13 19:22:36 +01:00
Kim De Mey 36d430aaa2
Add headers with proof content type and use it for verification (#1281)
* Add headers with proof content type and use it for verification

- Add BlockHeaderWithProof content type & content
- Use BlockHeaderWithProof content to verify if chain data is
part of the canonical chain
- Adjust parser & seeder code to be able to seed these headers
with proof
- Adjust eth_data_exporter to be able to export custom header
ranges for which to build proofs (mostly for testing)

There is currently quite some ugliness & clean-up needed for which
a big part is due tos upporting both BlockHeader and
BlockHeaderWithProof on the network.

* Change accumulator proof to array / SSZ vector type

- Change accumulator proof to SSZ vector instead of SSZ list.
- Add and use general buildProof and buildHeaderWithProof func.

* Make the BlockHeaderWithProof an SSZ Union with None option

* Update portal-spec-tests to master commit
2022-11-04 09:27:01 +01:00
Kim De Mey 9430619272
Remove Accumulator as network content type and add it to binary (#1267)
Portal master accumulator was removed from the network specs as a
content type shared on the network, as since the merge this is
a finite accumulator (pre-merge only).
So in this PR the accumulator gets removed as network type and
gets instead baked into the library. Building it is done by
seperate tooling (eth_data_exporter).
Because of this a lot of extra code can be removed that was
located in history_network, content_db, portal_protocol, etc.

Also removed to option to build the accumulator at start-up
of fluffy as this takes several minutes making it not viable.
It can still be loaded from a provided file however.

The ssz accumulator file is for now stored in the recently
created portal-spec-tests repository.
2022-10-17 20:38:51 +02:00
Kim De Mey 4e08f774d5
Fix issue where invalid / not validated would get gossiped (#1247)
Also requires us to split header data propagation from block body
and receipts propagation as the now fixed bug would allow for more
data to be gossiped even when data does not get validated (which
requires the headers).
2022-09-29 08:42:54 +02:00
Kim De Mey 011f44abea
Split parsing and seeding functionality for history data (#1218) 2022-09-09 21:21:48 +02:00
KonradStaniec f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec a1253c67bd
Fix testnet tests (#1176)
* Testnet improvements

Increase timeout for reading
Add more logs
Offer endpoint can fail due to talkReq timeout, to avoid
test failure, retry it few times until success.
2022-07-29 14:24:07 +02:00
KonradStaniec e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
Kim De Mey cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey 49bdaa112e
Increase wait time for content propagation to avoid CI failures (#1122) 2022-06-14 14:23:07 +02:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
Kim De Mey 553be51217
Change radius-config option to radius and allow also logRadius (#1100) 2022-05-23 23:23:24 +02:00
KonradStaniec 4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
Kim De Mey 69366e1880
Improve selecting of nodes in neighborhood gossip (#1072)
Allow also concurrent neighborhood gossip jobs when seeding data
into the network.
Update Grafana dashboard for two additional metrics regarding
lookups in neighborhood gossip.
2022-05-07 13:50:16 +02:00
Kim De Mey 0fba19b81a
Improvements to the propagation and seeding of data (#1058)
* Improvements to the propagation and seeding of data

- Use a lookup for nodes selection in neighborhoodGossip
- Rework populate db code and add `propagateBlockHistoryDb` call
and portal_history__propagateBlock json-rpc call
- Small adjustment to blockwalk

* Avoid storing out-of-range data in the propagate db calls
2022-04-12 15:49:19 +02:00
Kim De Mey 6f21c232ae
Add block bodies to the propagation and lookups (#975)
* Add block bodies to the propagation and lookups

- Read and propagate block bodies next to the headers
- Add block bodies content (via lookups) to the eth_getBlockByHash
call
- Test the above in test_portal_testnet

* Fix storage/propagation of block bodies

- Data format is an actual block: [header, txs, uncles], which
requires some adjustment to store the block body
- Added also `eth_getBlockTransactionCountByHash` json rpc call
2022-02-22 11:52:44 +01:00
Kim De Mey bd364ba2a6
Close json-rpc http client after each call in portal_tesnet test (#972)
This is to avoid having a json rpc call fail if there was a
previous call done more than 10 seconds ago. 10 seconds because
that is the default timeout on the http server side.
2022-02-16 22:06:43 +01:00
Kim De Mey f29e307fd0
Add radius cli option to Fluffy and use it in local testnet script (#968) 2022-02-15 13:11:27 +01:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
Kim De Mey 38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey 81ebfd2b2a
Add Portal protocol config and add config options to fluffy cli (#928) 2022-01-18 09:01:22 +01:00
Kim De Mey 70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00
Kim De Mey 2ea35ef82a
Add resolve call for Portal networks (#902)
* Add resolve call for Portal networks

And:
- Refactor some code by adding a findNodeVerified call
- add the portal network lookup json-rpc call that uses resolve
- add usage of this lookup in the portal testnet tests
- Additional comments

* Let recordsFromBytes fail on invalid ENRs

This behaviour is more similar as how it is done in discovery v5
base layer.
2021-12-08 09:26:31 +01:00
Kim De Mey 63b0945b52
Use json rpc client to run tests on portal testnet (#899)
Tests have also been added & adjusted. Original ones were actually
failing but unnoticed due to mistake in port passed in curl command.
2021-12-03 09:51:25 +01:00
Kim De Mey 0072089909
Implement part of the discv5 json-rpc calls (#894)
* Implement part of the discv5 json-rpc calls

* Use json marshalling for NodeId and Record everywhere
2021-11-29 10:39:37 +01:00