Commit Graph

218 Commits

Author SHA1 Message Date
Kim De Mey 011f44abea
Split parsing and seeding functionality for history data (#1218) 2022-09-09 21:21:48 +02:00
Kim De Mey 621c6a31a7
Fix bug in inCurrentEpoch and improve accumulator related tests (#1217)
* Fix bug in inCurrentEpoch and improve accumulator related tests

- Fix negative wraparound / underflow in inCurrentEpoch
- Add tests in accumulator tests to verify the above
- Add header offer tests with accumulator that does and doesn't
contain historical epochs
- Additional clean-up of history tests
- enable canonicalVerify in the tests
2022-09-09 13:12:09 +02:00
Jacek Sieka c2ed731fa5
eth: adapt to smaller eth_types (#1210) 2022-09-03 20:15:35 +02:00
Kim De Mey 5355f4e73a
A bit of consistency clean-up in accumulator code (#1201) 2022-08-24 22:12:56 +02:00
Kim De Mey c0a6c9a75c
Remove the usage of AccumulatorDB in the accumulator tests (#1200) 2022-08-24 16:41:04 +02:00
Kim De Mey 55dcbe9e58
Make the default network for portalcli the history network (#1195) 2022-08-23 16:28:10 +02:00
KonradStaniec 88263e664b
Clean up (#1197) 2022-08-22 12:23:26 +02:00
KonradStaniec 5f2117d081
Refactor portal stream (#1191) 2022-08-17 09:32:06 +02:00
KonradStaniec c5101c16e9
Bump nim eth (#1188)
* Bump nim eth

* Bump actions ubuntu
2022-08-16 09:17:57 +02:00
KonradStaniec f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
KonradStaniec 4f8d10f338
Bump nim-eth (#1185) 2022-08-06 10:11:36 +02:00
KonradStaniec 71f9e37482
Add getting block by number through accumulator (#1182) 2022-08-04 08:34:53 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec a1253c67bd
Fix testnet tests (#1176)
* Testnet improvements

Increase timeout for reading
Add more logs
Offer endpoint can fail due to talkReq timeout, to avoid
test failure, retry it few times until success.
2022-07-29 14:24:07 +02:00
KonradStaniec e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey 5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey facd7342fb
Add network retries for fluffy block requests (#1153)
- Add retries when the network request failed on validation of
block header, body or receipts.
- Shuffle the first set of neighbours that the request is send to
in order to not always hit the same peer first for the same content
- Some general clean-up
2022-07-05 14:42:55 +02:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
Kim De Mey cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Kim De Mey bf511b091e
Docs: Add separate section for seeding history data (#1140)
- Move section of eth_data_exporter out of default way to run
Fluffy
- Update section to explain full flow of seeding history data
into the network
2022-06-23 14:14:31 +02:00
Kim De Mey 589ba8367e
Adjust Grafana dashboard to represent the one used for the fleet (#1139)
This means adding the instance and container variables which can
be used to select the metrics of the specific fleet node.
2022-06-22 17:22:07 +02:00
KonradStaniec 196428dcf5
Use one ws connection to transfer data from source (#1136) 2022-06-22 08:50:58 +02:00
KonradStaniec 489b975189
Add seed db file format (#1126)
* Add seed db file format
2022-06-20 16:52:48 +02:00
Kim De Mey de2051dcf8
Add Accumulator build helper calls and refactors (#1127)
- Move the accumulator definitions to a history accumulator file
- Add accumulator build helper calls + temporary database
- Add a header gossip content key encoding test
- Refactor & some cleanup
2022-06-16 08:50:29 +02:00
Kim De Mey 6325712aa6
Add accumulator content keys for history network (#1123) 2022-06-14 23:38:34 +02:00
Kim De Mey 49bdaa112e
Increase wait time for content propagation to avoid CI failures (#1122) 2022-06-14 14:23:07 +02:00
KonradStaniec b23789d107
Add getting receipts from history network (#1118)
* Add getting receipts from history network
2022-06-10 12:24:53 +02:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
KonradStaniec 0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
Kim De Mey 6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
Kim De Mey 7216160ad0
Update Ultralight bootstrap nodes for testnet (#1110) 2022-06-01 08:52:54 +02:00
KonradStaniec 5c78fe64e4
Fix utp connection leak on cancel (#1107)
* Fix utp connection leak on cancel
2022-05-31 14:28:02 +02:00
KonradStaniec af10e8f179
Do not use vacuum when pruning (#1103)
* Do not use vacuum when pruning
2022-05-26 08:26:08 +02:00
Kim De Mey f219c69840
Rework of Portal json-rpc debug API and related functionality (#1102)
- More consistent naming in calls used by the JSON-RPC debug API
- Use storeContent everywhere to make sure content is only stored
if in range
- Remove populateHistoryDb subcommand and replace by JSON-RPC call
storeContent
- Remove some whitespace and fix some indentations
2022-05-24 13:27:22 +02:00
Kim De Mey 553be51217
Change radius-config option to radius and allow also logRadius (#1100) 2022-05-23 23:23:24 +02:00
Kim De Mey a69b16abff
Cleanup some imports and fix some warnings in the process (#1096) 2022-05-22 22:44:15 +02:00
Kim De Mey 6e05c7588e
Add Header Accumulator for the Header Gossip Network (#1094) 2022-05-18 22:49:35 +02:00
KonradStaniec 6a9d875fe7
Add pruning metrics (#1093) 2022-05-17 12:47:24 +02:00
Kim De Mey 56c3e31efc
Refactor and clean-up of Portal wire protocol test (#1089) 2022-05-16 13:17:42 +02:00
KonradStaniec 4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
KonradStaniec d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey 69366e1880
Improve selecting of nodes in neighborhood gossip (#1072)
Allow also concurrent neighborhood gossip jobs when seeding data
into the network.
Update Grafana dashboard for two additional metrics regarding
lookups in neighborhood gossip.
2022-05-07 13:50:16 +02:00
Kim De Mey 7bd9afa465
Bump nim-eth for max discv5 packet size (#1070) 2022-05-03 09:18:33 +02:00
Kim De Mey 99eea47dff
Fix clean-up socket in FindContent on cancel (#1067) 2022-04-20 18:46:00 +02:00
Kim De Mey 991edc1711
Fix log for block body network fetching (#1066) 2022-04-15 16:10:05 +02:00
Kim De Mey f37dde1177
Fix two possible index errors on Portal wire protocol (#1061) 2022-04-13 17:42:32 +02:00
Kim De Mey 3d818ad8ee
Update Fluffy README.md with metrics information 2022-04-13 16:11:01 +02:00
Kim De Mey 9626655557
Few more log improvements for Fluffy (#1060) 2022-04-13 11:17:07 +02:00
Kim De Mey d337806301
Improve logs and loglevels, and general cleanup (#1059) 2022-04-13 07:56:01 +02:00
Kim De Mey 0fba19b81a
Improvements to the propagation and seeding of data (#1058)
* Improvements to the propagation and seeding of data

- Use a lookup for nodes selection in neighborhoodGossip
- Rework populate db code and add `propagateBlockHistoryDb` call
and portal_history__propagateBlock json-rpc call
- Small adjustment to blockwalk

* Avoid storing out-of-range data in the propagate db calls
2022-04-12 15:49:19 +02:00
Kim De Mey 0195d5830b
Fix portal_content_keys_offered metric regression (#1057) 2022-04-12 10:59:41 +02:00
Kim De Mey ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
Kim De Mey 82aab2a404
Bump chronos and json-rpc and avoid http conn re-use for blockwalk (#1053)
Currently re-using http connections for the json-rpc requests
seems to slow down the requests considerably. Avoid this by
forcing a close after each requests in the blockwalk tool.
2022-04-11 19:39:45 +02:00
KonradStaniec eb84eb0854
Improve content offer (#1050)
* Improve content offer
2022-04-11 11:25:36 +02:00
KonradStaniec 0365d030fb
Increase packet size of utp over discv5 (#1042)
* Increase packet size of utp over discv5
2022-04-07 13:54:49 +02:00
KonradStaniec 203d8e8b70
Inital poke implementation (#1037)
* Initial poke implementation
2022-04-06 13:47:23 +02:00
KonradStaniec b3570fae6a
Add api to get n furthest elements from db (#1026)
* Add api to get n furthest elements from db
2022-04-03 15:14:44 +02:00
Kim De Mey 6a90937d4b
Add Trin and Ultralight bootstrap nodes to Fluffy (#1028) 2022-04-01 20:37:38 +02:00
Kim De Mey 9d656e99c4
Add concurrency to the content offers of neighborhoodGossip proc (#1027)
* Add concurrency to the content offers of neighborhoodGossip proc

And remove some whitespace

* Remove more whitespace and adjust for 80 char line limit

* Update fluffy grafana dashboard to include gossip offer results
2022-04-01 18:01:50 +02:00
Kim De Mey 8c3f210526
Add blockwalk tool to test range of blocks on Portal network (#1020) 2022-03-30 08:55:38 +02:00
KonradStaniec 4cb4f50bf3
Refactor connection process (#1016)
* Refactor connection process
2022-03-30 07:12:39 +02:00
Kim De Mey 392a540eb5
Remove unused import of config to avoid select_backend db import (#1018)
* Remove unused import of config to avoid select_backend db import

- Importing nimbus-eth1 config.nim causes import of select_backend
which will default cause an import of kvstore_rocksdb and thus a
require rocksdb. Remove unused one to avoid rocksdb dependency
for Fluffy.
- Remove some whitespace in bridge_client (to make fluffy CI
trigger for sure).

* Use specific cache keys for fluffy CI workflow

* Disable Fluffy CI reproducibility test
2022-03-29 15:51:48 +02:00
Kim De Mey 97c0cea767
Add grafana dashboard for Fluffy metrics (#1012) 2022-03-28 09:43:55 +02:00
Kim De Mey 89023782f3
Let Portal wire protocol adhere to the discv5 packet size limits (#1008)
- Truncate returned number of ENRs in Portal Nodes message to fit
the discv5 max. packet size
- Truncate returned number of ENRs in Portal Content message to
fit the discv5 max. packet size
- Use more detailed packet size calculation for max content payload
in Portal Content message.
- Add metrics to track how many ENRs get packed in the Portal
Nodes and Content messages.
2022-03-24 21:06:13 +01:00
Kim De Mey bde07054d1
Add metrics to the Portal wire protocol (#1006) 2022-03-23 16:32:59 +01:00
Kim De Mey 74127644db
Adjust findNodes proc naming for consistency (#1002) 2022-03-19 08:54:42 +01:00
Kim De Mey aff785ab48
Avoid opening an uTP stream when no content was accepted (#1001)
When the returned content key bitlist of an accept message is all
zeroes, neither side should try to set up an uTP stream.
2022-03-18 18:29:06 +01:00
Kim De Mey 08aa20a791
Fix error when dataDir does not exist (#1000)
- When dataDir does not exist, attempt to create the dir
- Some additional logging improvements
- Whitespace clean-up
2022-03-18 13:06:57 +01:00
Kim De Mey d3b0254d15
Update fluffy README with docs on testnet0 (#999) 2022-03-17 22:41:43 +01:00
Kim De Mey ae4aef6065
Add bootstrap nodes at compile time for testnet0 (#998)
- Add bootstrap nodes from a bootstrap_nodes.txt file at compile
time
- Add --network flag to select network specific bootstrap nodes
2022-03-17 18:39:24 +01:00
Kim De Mey b14dfea553
Improve portal wire logging by adding protocolId (#997) 2022-03-17 13:19:36 +01:00
KonradStaniec a330f0a4ef
Add api to get db size to content db (#987)
* Add size proc to content_db

* Add way to reclaim used space
2022-03-08 14:49:41 +01:00
jangko 9a1c8fc779
refactor genesis.nim 2022-03-03 08:28:00 +07:00
KonradStaniec b951139af9
Add basic validation of history network (#983)
* Add basic validation of history network

* Save only valid objects to db
2022-03-02 15:29:12 +01:00
Kim De Mey f8ede9659e
Fix runtime loglevel selection for Fluffy (#980) 2022-02-24 22:50:59 +01:00
Kim De Mey bc63f99ea6
Update local_testnet.md (#976)
More information on how the local testnet startup and the testing of it works.
2022-02-22 15:55:47 +01:00
Kim De Mey 6f21c232ae
Add block bodies to the propagation and lookups (#975)
* Add block bodies to the propagation and lookups

- Read and propagate block bodies next to the headers
- Add block bodies content (via lookups) to the eth_getBlockByHash
call
- Test the above in test_portal_testnet

* Fix storage/propagation of block bodies

- Data format is an actual block: [header, txs, uncles], which
requires some adjustment to store the block body
- Added also `eth_getBlockTransactionCountByHash` json rpc call
2022-02-22 11:52:44 +01:00
Kim De Mey d1127d77b1
Add cli option to start fluffy with a netkey file (#974)
Default the network key will be taken from a network key file
instead of randomly generated on each run. This is done because
the data that gets stored in the content database is dependant on
the network key used, as the node id is derived from this.
2022-02-17 14:13:39 +01:00
Kim De Mey bd364ba2a6
Close json-rpc http client after each call in portal_tesnet test (#972)
This is to avoid having a json rpc call fail if there was a
previous call done more than 10 seconds ago. 10 seconds because
that is the default timeout on the http server side.
2022-02-16 22:06:43 +01:00
Kim De Mey f29e307fd0
Add radius cli option to Fluffy and use it in local testnet script (#968) 2022-02-15 13:11:27 +01:00
jangko 6374c9f66d
fixes regression caused by #962 2022-02-14 21:22:39 +07:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
KonradStaniec 20068de838
Add more utp test cases (#957)
* Add more utp test cases
2022-02-07 14:19:57 +01:00
KonradStaniec a20917a547
Add docker file for fluffy (#954) 2022-02-07 09:03:19 +01:00
KonradStaniec 6869eafce6
Optimize size of utp-test-app docker file (#955) 2022-02-04 14:49:29 +01:00
KonradStaniec cf249109fa
Add utp test to ci (#952) 2022-02-03 13:11:54 +01:00
Kim De Mey 38036966a6
Improve the tests of the local testnet (#953)
* Improve the tests of the local testnet

The local testnet test was rather flaky and would occasionally
fail. It has been made more robust by adding the ENRs directly
to the routing table instead of doing some random lookups.

Additionally, the amount of nodes were increased (=64), ip limits
configuration was added, and the bits-per-hop value was set to 1
in order to make the lookups more likely to hit the network
instead of only the local routing table.

Failure is obviously still possible to happen when sufficient
packets get lost. If this turns out to be the case with the current
amount of nodes, we might have to revise the testing strategy here.

* Disable lookup test for State network

Disable lookup test for State network due to issue with custom
distance function causing the lookup to not always converging
towards the target.
2022-02-02 22:48:33 +01:00
Kim De Mey ac1196d915
Bump nim-ssz-serialization and clean-up fromSszBytes raises (#948) 2022-01-31 21:57:34 +01:00
KonradStaniec a8fdf8ec36
Improve utp test app (#947)
* Improve utp test app
2022-01-31 18:40:00 +01:00
KonradStaniec 990846135a
Bump nim-eth (#941)
* Bump nim-eth
2022-01-25 10:19:16 +01:00
Kim De Mey d2f56463ce
Remove the use of ByteList in PortalStream and for FoundContent (#937) 2022-01-20 21:21:20 +01:00
Kim De Mey 3909675f29
Use socket destroy instead of close to avoid extra FIN send (#935)
Also add missing push raises Defect
2022-01-20 15:04:23 +01:00
KonradStaniec cccd1e4640
Add tooling necessary to improve utp-testing (#931)
* Add tooling necessary to improve utp-testing
2022-01-19 16:06:23 +01:00
Kim De Mey 69b0cc07b6
Add data radius cache per portal protocol (#930) 2022-01-19 15:56:14 +01:00
Kim De Mey 14dd763900
Add populate history db option to fluffy (#929) 2022-01-18 15:08:02 +01:00