Commit Graph

170 Commits

Author SHA1 Message Date
Kim De Mey 99eea47dff
Fix clean-up socket in FindContent on cancel (#1067) 2022-04-20 18:46:00 +02:00
Kim De Mey 991edc1711
Fix log for block body network fetching (#1066) 2022-04-15 16:10:05 +02:00
Kim De Mey f37dde1177
Fix two possible index errors on Portal wire protocol (#1061) 2022-04-13 17:42:32 +02:00
Kim De Mey 3d818ad8ee
Update Fluffy README.md with metrics information 2022-04-13 16:11:01 +02:00
Kim De Mey 9626655557
Few more log improvements for Fluffy (#1060) 2022-04-13 11:17:07 +02:00
Kim De Mey d337806301
Improve logs and loglevels, and general cleanup (#1059) 2022-04-13 07:56:01 +02:00
Kim De Mey 0fba19b81a
Improvements to the propagation and seeding of data (#1058)
* Improvements to the propagation and seeding of data

- Use a lookup for nodes selection in neighborhoodGossip
- Rework populate db code and add `propagateBlockHistoryDb` call
and portal_history__propagateBlock json-rpc call
- Small adjustment to blockwalk

* Avoid storing out-of-range data in the propagate db calls
2022-04-12 15:49:19 +02:00
Kim De Mey 0195d5830b
Fix portal_content_keys_offered metric regression (#1057) 2022-04-12 10:59:41 +02:00
Kim De Mey ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
Kim De Mey 82aab2a404
Bump chronos and json-rpc and avoid http conn re-use for blockwalk (#1053)
Currently re-using http connections for the json-rpc requests
seems to slow down the requests considerably. Avoid this by
forcing a close after each requests in the blockwalk tool.
2022-04-11 19:39:45 +02:00
KonradStaniec eb84eb0854
Improve content offer (#1050)
* Improve content offer
2022-04-11 11:25:36 +02:00
KonradStaniec 0365d030fb
Increase packet size of utp over discv5 (#1042)
* Increase packet size of utp over discv5
2022-04-07 13:54:49 +02:00
KonradStaniec 203d8e8b70
Inital poke implementation (#1037)
* Initial poke implementation
2022-04-06 13:47:23 +02:00
KonradStaniec b3570fae6a
Add api to get n furthest elements from db (#1026)
* Add api to get n furthest elements from db
2022-04-03 15:14:44 +02:00
Kim De Mey 6a90937d4b
Add Trin and Ultralight bootstrap nodes to Fluffy (#1028) 2022-04-01 20:37:38 +02:00
Kim De Mey 9d656e99c4
Add concurrency to the content offers of neighborhoodGossip proc (#1027)
* Add concurrency to the content offers of neighborhoodGossip proc

And remove some whitespace

* Remove more whitespace and adjust for 80 char line limit

* Update fluffy grafana dashboard to include gossip offer results
2022-04-01 18:01:50 +02:00
Kim De Mey 8c3f210526
Add blockwalk tool to test range of blocks on Portal network (#1020) 2022-03-30 08:55:38 +02:00
KonradStaniec 4cb4f50bf3
Refactor connection process (#1016)
* Refactor connection process
2022-03-30 07:12:39 +02:00
Kim De Mey 392a540eb5
Remove unused import of config to avoid select_backend db import (#1018)
* Remove unused import of config to avoid select_backend db import

- Importing nimbus-eth1 config.nim causes import of select_backend
which will default cause an import of kvstore_rocksdb and thus a
require rocksdb. Remove unused one to avoid rocksdb dependency
for Fluffy.
- Remove some whitespace in bridge_client (to make fluffy CI
trigger for sure).

* Use specific cache keys for fluffy CI workflow

* Disable Fluffy CI reproducibility test
2022-03-29 15:51:48 +02:00
Kim De Mey 97c0cea767
Add grafana dashboard for Fluffy metrics (#1012) 2022-03-28 09:43:55 +02:00
Kim De Mey 89023782f3
Let Portal wire protocol adhere to the discv5 packet size limits (#1008)
- Truncate returned number of ENRs in Portal Nodes message to fit
the discv5 max. packet size
- Truncate returned number of ENRs in Portal Content message to
fit the discv5 max. packet size
- Use more detailed packet size calculation for max content payload
in Portal Content message.
- Add metrics to track how many ENRs get packed in the Portal
Nodes and Content messages.
2022-03-24 21:06:13 +01:00
Kim De Mey bde07054d1
Add metrics to the Portal wire protocol (#1006) 2022-03-23 16:32:59 +01:00
Kim De Mey 74127644db
Adjust findNodes proc naming for consistency (#1002) 2022-03-19 08:54:42 +01:00
Kim De Mey aff785ab48
Avoid opening an uTP stream when no content was accepted (#1001)
When the returned content key bitlist of an accept message is all
zeroes, neither side should try to set up an uTP stream.
2022-03-18 18:29:06 +01:00
Kim De Mey 08aa20a791
Fix error when dataDir does not exist (#1000)
- When dataDir does not exist, attempt to create the dir
- Some additional logging improvements
- Whitespace clean-up
2022-03-18 13:06:57 +01:00
Kim De Mey d3b0254d15
Update fluffy README with docs on testnet0 (#999) 2022-03-17 22:41:43 +01:00
Kim De Mey ae4aef6065
Add bootstrap nodes at compile time for testnet0 (#998)
- Add bootstrap nodes from a bootstrap_nodes.txt file at compile
time
- Add --network flag to select network specific bootstrap nodes
2022-03-17 18:39:24 +01:00
Kim De Mey b14dfea553
Improve portal wire logging by adding protocolId (#997) 2022-03-17 13:19:36 +01:00
KonradStaniec a330f0a4ef
Add api to get db size to content db (#987)
* Add size proc to content_db

* Add way to reclaim used space
2022-03-08 14:49:41 +01:00
jangko 9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
KonradStaniec b951139af9
Add basic validation of history network (#983)
* Add basic validation of history network

* Save only valid objects to db
2022-03-02 15:29:12 +01:00
Kim De Mey f8ede9659e
Fix runtime loglevel selection for Fluffy (#980) 2022-02-24 22:50:59 +01:00
Kim De Mey bc63f99ea6
Update local_testnet.md (#976)
More information on how the local testnet startup and the testing of it works.
2022-02-22 15:55:47 +01:00
Kim De Mey 6f21c232ae
Add block bodies to the propagation and lookups (#975)
* Add block bodies to the propagation and lookups

- Read and propagate block bodies next to the headers
- Add block bodies content (via lookups) to the eth_getBlockByHash
call
- Test the above in test_portal_testnet

* Fix storage/propagation of block bodies

- Data format is an actual block: [header, txs, uncles], which
requires some adjustment to store the block body
- Added also `eth_getBlockTransactionCountByHash` json rpc call
2022-02-22 11:52:44 +01:00
Kim De Mey d1127d77b1
Add cli option to start fluffy with a netkey file (#974)
Default the network key will be taken from a network key file
instead of randomly generated on each run. This is done because
the data that gets stored in the content database is dependant on
the network key used, as the node id is derived from this.
2022-02-17 14:13:39 +01:00
Kim De Mey bd364ba2a6
Close json-rpc http client after each call in portal_tesnet test (#972)
This is to avoid having a json rpc call fail if there was a
previous call done more than 10 seconds ago. 10 seconds because
that is the default timeout on the http server side.
2022-02-16 22:06:43 +01:00
Kim De Mey f29e307fd0
Add radius cli option to Fluffy and use it in local testnet script (#968) 2022-02-15 13:11:27 +01:00
jangko 6374c9f66d
fixes regression caused by #962 2022-02-14 21:22:39 +07:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
KonradStaniec 20068de838
Add more utp test cases (#957)
* Add more utp test cases
2022-02-07 14:19:57 +01:00
KonradStaniec a20917a547
Add docker file for fluffy (#954) 2022-02-07 09:03:19 +01:00
KonradStaniec 6869eafce6
Optimize size of utp-test-app docker file (#955) 2022-02-04 14:49:29 +01:00
KonradStaniec cf249109fa
Add utp test to ci (#952) 2022-02-03 13:11:54 +01:00
Kim De Mey 38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey ac1196d915
Bump nim-ssz-serialization and clean-up fromSszBytes raises (#948) 2022-01-31 21:57:34 +01:00
KonradStaniec a8fdf8ec36
Improve utp test app (#947)
* Improve utp test app
2022-01-31 18:40:00 +01:00
KonradStaniec 990846135a
Bump nim-eth (#941)
* Bump nim-eth
2022-01-25 10:19:16 +01:00
Kim De Mey d2f56463ce
Remove the use of ByteList in PortalStream and for FoundContent (#937) 2022-01-20 21:21:20 +01:00
Kim De Mey 3909675f29
Use socket destroy instead of close to avoid extra FIN send (#935)
Also add missing push raises Defect
2022-01-20 15:04:23 +01:00
KonradStaniec cccd1e4640
Add tooling necessary to improve utp-testing (#931)
* Add tooling necessary to improve utp-testing
2022-01-19 16:06:23 +01:00
Kim De Mey 69b0cc07b6
Add data radius cache per portal protocol (#930) 2022-01-19 15:56:14 +01:00
Kim De Mey 14dd763900
Add populate history db option to fluffy (#929) 2022-01-18 15:08:02 +01:00
Kim De Mey 81ebfd2b2a
Add Portal protocol config and add config options to fluffy cli (#928) 2022-01-18 09:01:22 +01:00
Kim De Mey f19a64a2fe
Add uTP over discv5 in portal wire protocol (#922)
* Add uTP over discv5 in portal wire protocol
2022-01-14 16:07:14 +01:00
Kim De Mey 79647e5580
Allow access to contentDB from portal wire protocol (#920)
- Allow access to contentDB from portal wire protocol
- Use this to do the db.get in `handleFindContent` directly
- Use this to check the `contentKeys` list in `handleOffer`
2022-01-06 09:06:05 +01:00
Kim De Mey a8640fe57c
History content keys (#919)
* Change History content key to us SSZ Union and adjust tests

* Change slot to byteBE instead of LE

This is currently not specified in the Portal network
specifications, but we are using already BE for the actual content
key, so change it also here to remain consistent.
2022-01-05 09:49:49 +01:00
Kim De Mey f4ca598c03
Some version of the findContent JSON-RPC proc (#918)
Also includes fixing instantiation of the history network json-rpc
API handles.
2021-12-20 11:57:55 +01:00
Kim De Mey 796787e0e5
Add state and history content key tests (#916) 2021-12-20 09:53:00 +01:00
Kim De Mey 092d946c72
Add state ContentKey test and adjust history ContentKey test (#914) 2021-12-15 08:48:56 +01:00
Kim De Mey f3d0e97997
Add Portal protocol testing docs and more (#912)
- Add portal network ping and findNodes JSON-RPC endpoints
- clean-up of some cli arguments
- Add protocol_interop.md document and adjust/fix other
documentation
2021-12-13 14:12:51 +01:00
Kim De Mey 70625bc02c
Use one bootstrap nodes argument for discv5 and Portal (#911)
Currently bootstrap nodes for discv5 and for the Portal nodes
were provided through separate cli arguments. This is however
confusing and cumbersome as typically when (currently) testing
a node will have both discv5 and the Portal networks enabled.
We merge them into one argument.

If a node happens not to support a Portal network, it will be
removed after message request failure.

Commit also contains additional clean-up and nim-eth bump.
2021-12-13 09:06:29 +01:00
Kim De Mey 10a3946194
Bit of cleanup (#903)
* Rename FindNode to FindNodes as per spec

* Use consistently lower case starting camelCase for consts

Style guide nep1 allows for both CamelCase and camelCase for
consts, but we seem to use more often camelCase. Using this now
consistently.

* Change some procs to func
2021-12-08 11:54:22 +01:00
Kim De Mey 2ea35ef82a
Add resolve call for Portal networks (#902)
* Add resolve call for Portal networks

And:
- Refactor some code by adding a findNodeVerified call
- add the portal network lookup json-rpc call that uses resolve
- add usage of this lookup in the portal testnet tests
- Additional comments

* Let recordsFromBytes fail on invalid ENRs

This behaviour is more similar as how it is done in discovery v5
base layer.
2021-12-08 09:26:31 +01:00
Kim De Mey 63b0945b52
Use json rpc client to run tests on portal testnet (#899)
Tests have also been added & adjusted. Original ones were actually
failing but unnoticed due to mistake in port passed in curl command.
2021-12-03 09:51:25 +01:00
Kim De Mey ed04aad203
Fix fluffy discovery rpc tests after nim-json-rpc bump (#897) 2021-11-30 16:04:24 +01:00
Kim De Mey 0072089909
Implement part of the discv5 json-rpc calls (#894)
* Implement part of the discv5 json-rpc calls

* Use json marshalling for NodeId and Record everywhere
2021-11-29 10:39:37 +01:00
Kim De Mey 5f4ea7822c
Have one generic installPortalApiHandlers to be used for different Portal networks (#893) 2021-11-24 14:53:01 +01:00
Kim De Mey 77cc9b6215
Add support for passing bootstrap file and write local enr file (#892)
Also use this now in the local testnet script instead of hardcoded
values.
2021-11-24 12:12:25 +01:00
Kim De Mey 903350bdde
Add local testnet script and required json-rpc calls (#891)
- Add basic discv5 and portal json-rpc calls and activate them in
fluffy
- Renames in the rpc folder
- Add local testnet script and run this script in CI
- bump nim-eth
2021-11-24 08:45:55 +01:00
Kim De Mey 78f585399f
Bump submodules (#887)
* Bump submodules

* Fix missing import for state and history networks
2021-11-18 12:06:53 +01:00
Kim De Mey bdbd8e4c87
Add SSZ Unions through case objects (#882)
* Add SSZ Unions through case objects

* Add connection id content response test and improve other test vectors

* Implement content keys and ids for state network as per spec

Content keys case object is used so that it can be serialized and
deserialized as an SSZ Union.

* Let message Union in Portal wire protocol start at 0 as per new spec
2021-11-17 17:11:17 +01:00
Kim De Mey 414fdafab9
Improve the populating of the routing table (#886)
- Search for the node on an incoming portal message and try to add
it to the routing table
- Don't rely on discv5 nodes for bootstrapping portal networks for
now.
- Attempt to repopulate the routing table when it is at 0 or drops
to 0.
2021-11-16 17:50:08 +01:00
Kim De Mey 0ba3c682e4
Use SSZ code from nim-ssz-serialization module (#875) 2021-10-23 14:28:12 +02:00
Kim De Mey 02e8b2eb12
Clean up old unused code and unused imports (#877) 2021-10-22 21:36:53 +02:00
Kim De Mey 92dfb02ad7
Use ByteList.init instead of List 2048 (#876) 2021-10-22 17:49:23 +02:00
Kim De Mey 4b498356fc
Improve portal wire logging on sending/receiving messages (#874) 2021-10-22 11:03:00 +02:00
Kim De Mey 4b85c73f43
Update portal wire protocol ids to latest spec (#873) 2021-10-20 22:31:05 +02:00
Kim De Mey b77034c870
Update portal wire protocol according to latest spec changes (#860)
This is not complete, in order to be according to specification
two SSZ Union types need to be used.
2021-10-13 21:35:54 +02:00
Kim De Mey 447deb29ff
Add HistoryNetwork that uses Portal wire protocol (#856)
* Add HistoryNetwork that uses Portal wire protocol

* Further implement getContent and adjust naming of distance calc calls
2021-10-09 13:22:03 +02:00
KonradStaniec c62905db25
Re enable using custom distance function (#854) 2021-10-05 21:16:33 +02:00
Kim De Mey b58920f29f
Set config default descriptions (#852)
* Update fluffy config to set defaults descriptions

* Update portalcli config to set defaults descriptions
2021-09-29 13:58:55 +02:00
Kim De Mey 51626c5831
Add a basic ContentDB for portal networks (#848)
* Add a basic ContentDB for Portal networks

* Use ContentDB in StateNetwork

* Avoid probably some form of sandwich problem by re-exporting kvstore_sqlite3 from content_db
2021-09-28 19:58:41 +02:00
Kim De Mey 8e26acc4eb
Move tools and add common types (#846)
* Move some duplicated types to common_types

* Move portalcli to tools/ and update the READMEs
2021-09-24 17:18:54 +02:00
Kim De Mey 16a36453b5
Rework ContentId, ContentKey, GetContent to be more clear (#845)
* Small clarification in naming and note where GetContent is actually needed

* Rework contentId and contentKey
2021-09-24 11:22:07 +02:00
Kim De Mey 785a3b47b0
Allow for passing Portal specific bootstrap nodes (#844)
* Allow for passing Portal specific bootstrap nodes

* Fix to also replaceNode when decodeMessage fails

* Add portal bootstrap node tests and reorder test cases
2021-09-23 14:26:41 +02:00
Kim De Mey 6f6345a022
Move Portal wire protocol out of state specific code (#843)
* Allow for custom protocol id in portal wire protocol

* Move Portal wire protocol out of state specific code
2021-09-22 17:07:14 +02:00
Kim De Mey 11c120e400
Update wire protocol of state network spec (#829)
* Update wire protocol of state network spec

* BytesN should be SSZ Vector not List

* Add specific lists and bitlists for content keys
2021-09-22 11:28:04 +02:00
Kim De Mey 0135189e4c
Oops, isMainModule will not run of course for all_tests (#841) 2021-09-21 18:38:08 +02:00
Kim De Mey 1afdfdf2ff
Disable mixed usage of cliBuilder to fix incorrect test failure reporting (#836) 2021-09-21 14:35:41 +02:00
Kim De Mey 7a42d037f2
Disable the custom distance functions for Portal Protocol (#832)
The Portal Network tests fail with the custom functions,
on one part due to the fact that the portal tests still
uses the xor functions, but also because the
`neighboursAtDistances` call appears to filter them out.
The reverse calculation might be off.
2021-09-20 12:55:03 +02:00
jangko a3badea928
config: fix new config based on input from jamie and zahary 2021-09-18 17:34:51 +07:00
KonradStaniec 6192cd7dc1
Use custom distance function in state network (#831)
* Use custom distance calculator in state network
2021-09-16 16:13:36 +02:00
KonradStaniec 8f683bd318
bump nim-eth module (#828)
* bump-nim-eth

* fix breaking changes

* Increase result limit
2021-09-15 11:24:03 +02:00
KonradStaniec 58b11e683d
Implement custom distance function (#827)
* Implement custom distance function

* More docs about function equivalence
2021-09-14 13:36:55 +02:00
Kim De Mey d34681d48f
Little nitpick clean-up (#826) 2021-09-13 15:56:44 +02:00
jangko bf87c6e83c
tests: fixes fluffy test due to recent changes
- CustomGenesis -> CustomNetwork
- loadCustomGenesis -> loadCustomNetwork
- BaseChainDB constructor changes
2021-09-09 08:09:35 +07:00
Kim De Mey 07b2116f6b
Another nim-eth bump and API changes fix (#821) 2021-09-07 15:56:51 +02:00
KonradStaniec 2c620c007d
Fix bugs in lookup and add test case (#820) 2021-09-07 14:22:25 +02:00
Kim De Mey 832a50df6e
Bump nim-eth module and adjust for `distanceTo` API change (#819) 2021-09-06 17:04:21 +02:00
KonradStaniec a083fb30fc
Generalize network layer for portal (#814)
* Generalize netork layer for portal

* Make messages free from any content references

* Use portal network in main fluffy module

* Fix cli

* Use lookup in portal network

* Avoid using result
2021-09-03 10:57:19 +02:00