Commit Graph

36 Commits

Author SHA1 Message Date
Kim De Mey 56caa5f62f
Add radius sort in trace lookup and logging improvements (#2905)
The radius sort performance improvement in content lookups was
not implemented in the trace version.

Also cleanup some part of the logging related to uTP connection
setup.
2024-12-04 13:59:08 +01:00
bhartnett 359eb6d974
Fluffy: Portal stream improvements and pending transfers prune fix (#2900)
* Fix defect in for loop when deleting element.

* Prune offers and requests before accepting.

* Store content requests and offers by connectionId.
2024-12-04 08:34:13 +08:00
Kim De Mey 0f18de61dc
Revert commit 6142183 and partial of b446d2a (#2898)
There is an assertion hitting due to the additon of an iterator
that deletes items from the sequence while iteratting over it.
Before the keepIf helper was used that has different code for
doing this similar work.
2024-12-02 14:09:58 +01:00
bhartnett b446d2a2e8
Fluffy: Cleanup state bridge preimages backend and add logging for transfer limit reached (#2896) 2024-12-02 16:09:43 +08:00
bhartnett 6142183d2c
Fluffy: Limit concurrent offers that can be received from each peer (#2885)
* Limit offer transfers per peer.

* Remove pending transfers in prune.

* Limit content lookups.

* Improve performance of canAddPendingTransfer and addPendingTransfer.
2024-11-29 11:31:46 +08:00
Kim De Mey 9c9a41a16d
More verbose error handling for Portal stream content reading (#2737) 2024-10-14 11:53:28 +02:00
Kim De Mey 4f0bc49a84
Remove the early-fin and add a 4s timeout on socket destroy instead (#2614) 2024-09-12 10:47:02 +02:00
Kim De Mey c31fc37c62
Add async raises annotations for Portal wire and Portal networks (#2361) 2024-06-14 14:21:30 +02:00
andri lim 5a18537450
Bump nim-eth, nim-web3, nimbus-eth2 (#2344)
* Bump nim-eth, nim-web3, nimbus-eth2

- Replace std.Option with results.Opt
- Fields name changes

* More fixes

* Fix Portal stream async raises and portal testnet Opt usage

* Bump eth + nimbus-eth2 + more fixes related to eth_types changes

* Fix in utp test app and nimbus-eth2 bump

* Fix test_blockchain_json rebase conflict

* Fix EVMC block_timestamp conversion plus commentary

---------

Co-authored-by: kdeme <kim.demey@gmail.com>
2024-06-14 14:31:08 +07:00
Kim De Mey 2d76b8c010
Formatting fluffy with nph v0.5.1 + CI check (#2020)
* Add nph check to fluffy CI lint

* Add a section on nph usage in the fluffy.guide

* Update copyright years for altered files

* Avoid chained methods formatting style in db code

* Update nph in CI to v0.5

* Remove leftover commented import

* Move comment to avoid nph turning complex list into simple list (nph bug)

* Update nph in CI to v0.5.1

* Formatting fluffy with nph v0.5.1
2024-02-28 18:31:45 +01:00
andri lim 56215ed83f
Bump stint to v2.0: new array backend (#1747)
* Bump stint to v2.0: new array backend
2023-09-13 09:32:38 +07:00
Daniel Sobol 69a1a988b0
don't gossip data back to the node we rcvd it from [#1227] (#1719) 2023-09-04 12:21:01 +02:00
Kim De Mey dd4da7491d
Remove yield usage from portal stream and protocol (#1602) 2023-06-15 14:32:33 +02:00
Kim De Mey fdad9cfd00
Move content offered log after socket close wait (#1530)
To be clear that the socket also got closed before this.
Also explain that on socket close, the actual socket still waits
for an ACK.
2023-04-04 17:36:57 +02:00
Kim De Mey 73e93f1f11
Reduce Nim 1.6 compiler warnings & hints
* Reduce Nim 1.6 compiler warnings/hints for Fluffy and Nimbus proxy

Mostly raises Defect removals, TaintedString removal and some
unnecessary imports.

Also updating the copyright years alongside.

* Further reduce Nim 1.6 compiler warnings/hints for Nimbus
2023-01-31 13:38:08 +01:00
KonradStaniec 88263e664b
Clean up (#1197) 2022-08-22 12:23:26 +02:00
KonradStaniec 5f2117d081
Refactor portal stream (#1191) 2022-08-17 09:32:06 +02:00
KonradStaniec f0cd340163
Adhere to transport limits in seed methods (#1186)
* Add means to limit offered content to fit talkreq

* Add test for history network limits

* Change seed method api to return num of items offered
2022-08-09 14:32:41 +02:00
Kim De Mey 50af402f59
Add verification of block data by use of accumulator (#1178) 2022-08-01 21:00:21 +02:00
KonradStaniec a1253c67bd
Fix testnet tests (#1176)
* Testnet improvements

Increase timeout for reading
Add more logs
Offer endpoint can fail due to talkReq timeout, to avoid
test failure, retry it few times until success.
2022-07-29 14:24:07 +02:00
Kim De Mey 5abf05e455
Add validation for offered block bodies and receipts (#1158)
Also use a AsyncQueue now to handle offered data instead of the
network specific validation callbacks.
2022-07-11 16:29:16 +02:00
Kim De Mey fb12793482
Bump nim-eth and nim-bearssl and accompanying fixes (#1150) 2022-07-04 09:38:02 +02:00
Kim De Mey cad74db423
Support for multiple content items over Portal stream (#1090)
Each content item send gets prefix with varint (max u32) allowing
to send multiple content items as requested by offer/accept flow.
2022-06-24 15:35:31 +02:00
KonradStaniec 5c78fe64e4
Fix utp connection leak on cancel (#1107)
* Fix utp connection leak on cancel
2022-05-31 14:28:02 +02:00
Kim De Mey 7bd9afa465
Bump nim-eth for max discv5 packet size (#1070) 2022-05-03 09:18:33 +02:00
Kim De Mey d337806301
Improve logs and loglevels, and general cleanup (#1059) 2022-04-13 07:56:01 +02:00
Kim De Mey ed38ed494f
Add validation handler to Portal wire protocol (#1055)
- Validation handler allows for network specific validation of
the content
- Added also pruning of the allowed connections on PortalStream
2022-04-11 19:42:38 +02:00
KonradStaniec 0365d030fb
Increase packet size of utp over discv5 (#1042)
* Increase packet size of utp over discv5
2022-04-07 13:54:49 +02:00
Kim De Mey 9d656e99c4
Add concurrency to the content offers of neighborhoodGossip proc (#1027)
* Add concurrency to the content offers of neighborhoodGossip proc

And remove some whitespace

* Remove more whitespace and adjust for 80 char line limit

* Update fluffy grafana dashboard to include gossip offer results
2022-04-01 18:01:50 +02:00
KonradStaniec 4cb4f50bf3
Refactor connection process (#1016)
* Refactor connection process
2022-03-30 07:12:39 +02:00
Kim De Mey 0060462dc0
Sharing block header data around in a Portal history network (PoC) (#960)
* Sharing block header data around in a Portal history network (PoC)

- Rework PortalStream to have an instance per PortalProtocol (this
needs to be improved eventually). Each instance uses the same
UtpDiscv5Protocol instance.
- Add processContent on receival of accepted data
- Add dumb neighborhoodGossip: dumb in the sense that it only
offers one piece of content at a time.
- Add to / adjust populate_db to also allow for propagation of
the data and add debug rpc call: portal_history_propagate
- Add eth_rpc_client
- Add eth_getBlockbyHash (no txs or uncles) to eth API
- Add additional test to test_portal_testnet which loads 5 block
headers to 1 node, and offers this data to few nodes, which should
propagate it over the network further. Next query every node for
this data.

* Adjust paths on which Fluffy CI is triggered

* Add documentation on the local testnet
2022-02-11 14:43:10 +01:00
KonradStaniec 990846135a
Bump nim-eth (#941)
* Bump nim-eth
2022-01-25 10:19:16 +01:00
Kim De Mey d2f56463ce
Remove the use of ByteList in PortalStream and for FoundContent (#937) 2022-01-20 21:21:20 +01:00
Kim De Mey 3909675f29
Use socket destroy instead of close to avoid extra FIN send (#935)
Also add missing push raises Defect
2022-01-20 15:04:23 +01:00
Kim De Mey 81ebfd2b2a
Add Portal protocol config and add config options to fluffy cli (#928) 2022-01-18 09:01:22 +01:00
Kim De Mey f19a64a2fe
Add uTP over discv5 in portal wire protocol (#922)
* Add uTP over discv5 in portal wire protocol
2022-01-14 16:07:14 +01:00