Commit Graph

155 Commits

Author SHA1 Message Date
KonradStaniec a1253c67bd
Fix testnet tests (#1176)
* Testnet improvements

Increase timeout for reading
Add more logs
Offer endpoint can fail due to talkReq timeout, to avoid
test failure, retry it few times until success.
2022-07-29 14:24:07 +02:00
KonradStaniec e3cabaff7f
Add bulk seeding to multiple peers (#1170)
* Add bulk seeding to multiple peers
2022-07-26 13:14:56 +02:00
KonradStaniec aa945f1ed9
Additional seed endpoints (#1164)
* Additional seed endpoints
2022-07-20 12:46:42 +02:00
Kim De Mey 5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey f29cbebb9d
Move block data test files to one location (#1154) 2022-07-05 16:14:52 +02:00
Kim De Mey facd7342fb
Add network retries for fluffy block requests (#1153)
- Add retries when the network request failed on validation of
block header, body or receipts.
- Shuffle the first set of neighbours that the request is send to
in order to not always hit the same peer first for the same content
- Some general clean-up
2022-07-05 14:42:55 +02:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey 33015ea4a3
Adjust to the newly specified SSZ types for Block Body and Receipts (#1147) 2022-07-01 21:51:51 +02:00
KonradStaniec fdb048ed21
Add getLogs implementation (#1143)
* Add eth_getLogs json-rpc endpoint in Fluffy and Nimbus
2022-06-29 17:44:08 +02:00
Kim De Mey cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
Kim De Mey d4d4e8c28f
Add accumulator buildProof, verifyProof and verifyHeader (#1132) 2022-06-23 21:00:59 +02:00
Kim De Mey bf511b091e
Docs: Add separate section for seeding history data (#1140)
- Move section of eth_data_exporter out of default way to run
Fluffy
- Update section to explain full flow of seeding history data
into the network
2022-06-23 14:14:31 +02:00
Kim De Mey 589ba8367e
Adjust Grafana dashboard to represent the one used for the fleet (#1139)
This means adding the instance and container variables which can
be used to select the metrics of the specific fleet node.
2022-06-22 17:22:07 +02:00
KonradStaniec 196428dcf5
Use one ws connection to transfer data from source (#1136) 2022-06-22 08:50:58 +02:00
KonradStaniec 489b975189
Add seed db file format (#1126)
* Add seed db file format
2022-06-20 16:52:48 +02:00
Kim De Mey de2051dcf8
Add Accumulator build helper calls and refactors (#1127)
- Move the accumulator definitions to a history accumulator file
- Add accumulator build helper calls + temporary database
- Add a header gossip content key encoding test
- Refactor & some cleanup
2022-06-16 08:50:29 +02:00
Kim De Mey 6325712aa6
Add accumulator content keys for history network (#1123) 2022-06-14 23:38:34 +02:00
Kim De Mey 49bdaa112e
Increase wait time for content propagation to avoid CI failures (#1122) 2022-06-14 14:23:07 +02:00
KonradStaniec b23789d107
Add getting receipts from history network (#1118)
* Add getting receipts from history network
2022-06-10 12:24:53 +02:00
KonradStaniec 730013bde3
Change test data and propagation format (#1115)
* Change test data and propagation format

* Add tool to download data from local node
2022-06-08 15:14:01 +02:00
KonradStaniec 0776f35e0c
Use queries with custom xor function (#1113)
* Use queries with custom xor function
2022-06-03 13:44:42 +02:00
Kim De Mey 6d8b25a5f5
Update header accumulator test to start from Mainnet genesis (#1111) 2022-06-01 15:21:22 +02:00
Kim De Mey 7216160ad0
Update Ultralight bootstrap nodes for testnet (#1110) 2022-06-01 08:52:54 +02:00
KonradStaniec 5c78fe64e4
Fix utp connection leak on cancel (#1107)
* Fix utp connection leak on cancel
2022-05-31 14:28:02 +02:00
KonradStaniec af10e8f179
Do not use vacuum when pruning (#1103)
* Do not use vacuum when pruning
2022-05-26 08:26:08 +02:00
Kim De Mey f219c69840
Rework of Portal json-rpc debug API and related functionality (#1102)
- More consistent naming in calls used by the JSON-RPC debug API
- Use storeContent everywhere to make sure content is only stored
if in range
- Remove populateHistoryDb subcommand and replace by JSON-RPC call
storeContent
- Remove some whitespace and fix some indentations
2022-05-24 13:27:22 +02:00
Kim De Mey 553be51217
Change radius-config option to radius and allow also logRadius (#1100) 2022-05-23 23:23:24 +02:00
Kim De Mey a69b16abff
Cleanup some imports and fix some warnings in the process (#1096) 2022-05-22 22:44:15 +02:00
Kim De Mey 6e05c7588e
Add Header Accumulator for the Header Gossip Network (#1094) 2022-05-18 22:49:35 +02:00
KonradStaniec 6a9d875fe7
Add pruning metrics (#1093) 2022-05-17 12:47:24 +02:00
Kim De Mey 56c3e31efc
Refactor and clean-up of Portal wire protocol test (#1089) 2022-05-16 13:17:42 +02:00
KonradStaniec 4e8c5f292e
Dynamic radius adjustments (#1079)
* Add config for node radius

* Adjust radius when deleting content
2022-05-12 18:04:37 +02:00
KonradStaniec d3f231e3da
Add put method which preserves max size (#1065)
* Add put method which preserves max size
2022-05-09 17:18:57 +02:00
Kim De Mey 69366e1880
Improve selecting of nodes in neighborhood gossip (#1072)
Allow also concurrent neighborhood gossip jobs when seeding data
into the network.
Update Grafana dashboard for two additional metrics regarding
lookups in neighborhood gossip.
2022-05-07 13:50:16 +02:00
Kim De Mey 7bd9afa465
Bump nim-eth for max discv5 packet size (#1070) 2022-05-03 09:18:33 +02:00
Kim De Mey 99eea47dff
Fix clean-up socket in FindContent on cancel (#1067) 2022-04-20 18:46:00 +02:00
Kim De Mey 991edc1711
Fix log for block body network fetching (#1066) 2022-04-15 16:10:05 +02:00
Kim De Mey f37dde1177
Fix two possible index errors on Portal wire protocol (#1061) 2022-04-13 17:42:32 +02:00
Kim De Mey 3d818ad8ee
Update Fluffy README.md with metrics information 2022-04-13 16:11:01 +02:00
Kim De Mey 9626655557
Few more log improvements for Fluffy (#1060) 2022-04-13 11:17:07 +02:00
Kim De Mey d337806301
Improve logs and loglevels, and general cleanup (#1059) 2022-04-13 07:56:01 +02:00
Kim De Mey 0fba19b81a
Improvements to the propagation and seeding of data (#1058)
* Improvements to the propagation and seeding of data

- Use a lookup for nodes selection in neighborhoodGossip
- Rework populate db code and add `propagateBlockHistoryDb` call
and portal_history__propagateBlock json-rpc call
- Small adjustment to blockwalk

* Avoid storing out-of-range data in the propagate db calls
2022-04-12 15:49:19 +02:00
Kim De Mey 0195d5830b
Fix portal_content_keys_offered metric regression (#1057) 2022-04-12 10:59:41 +02:00
Kim De Mey ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
Kim De Mey 82aab2a404
Bump chronos and json-rpc and avoid http conn re-use for blockwalk (#1053)
Currently re-using http connections for the json-rpc requests
seems to slow down the requests considerably. Avoid this by
forcing a close after each requests in the blockwalk tool.
2022-04-11 19:39:45 +02:00
KonradStaniec eb84eb0854
Improve content offer (#1050)
* Improve content offer
2022-04-11 11:25:36 +02:00
KonradStaniec 0365d030fb
Increase packet size of utp over discv5 (#1042)
* Increase packet size of utp over discv5
2022-04-07 13:54:49 +02:00
KonradStaniec 203d8e8b70
Inital poke implementation (#1037)
* Initial poke implementation
2022-04-06 13:47:23 +02:00
KonradStaniec b3570fae6a
Add api to get n furthest elements from db (#1026)
* Add api to get n furthest elements from db
2022-04-03 15:14:44 +02:00
Kim De Mey 6a90937d4b
Add Trin and Ultralight bootstrap nodes to Fluffy (#1028) 2022-04-01 20:37:38 +02:00