Commit Graph

126 Commits

Author SHA1 Message Date
diegomrsantos dc83a1e9b6
chore(formatting): format the whole codebase using nph 0.5.1 (#1118) 2024-06-11 17:18:06 +02:00
diegomrsantos 88e233db81
fix: Asynchronous task [sendMsgSlow()] was cancelled [FutureDefect] (#1094) 2024-05-07 15:44:14 +02:00
Jacek Sieka 84659af45b
avoid latency/copy when sending low-priority messages to fast peers (#1060) 2024-05-02 12:26:16 +02:00
Jacek Sieka aef44ed1ce
salt `idontwant` (#1090) 2024-05-02 12:18:55 +02:00
diegomrsantos 1a707e1264
feat: add max number of elements to non-prio queue (#1077) 2024-03-25 22:00:11 +01:00
Álex Cabeza Romero 458b0885dd
fix(issue-1052): Single topic for RPC Message (#1061) 2024-03-25 12:06:34 +01:00
Jacek Sieka 467b5b4f0c
avoid cancelling send future (#1075) 2024-03-20 10:54:32 +00:00
Jacek Sieka ae13a0d583
Send priority with queue fix (#1051)
Co-authored-by: Diego <diego@status.im>
2024-03-05 15:05:21 +00:00
Diego c5e4f8e12d
Revert "feat: message prioritization with immediate peer-published dispatch and queuing for other msgs (#1015)"
This reverts commit fe4ff79885.
2024-02-19 13:47:37 +01:00
diegomrsantos fe4ff79885
feat: message prioritization with immediate peer-published dispatch and queuing for other msgs (#1015) 2024-02-16 10:54:16 +01:00
diegomrsantos 5594bcb33e
fix: more metrics issues when libp2p_expensive_metrics is enabled (#1016) 2024-01-30 16:55:55 +01:00
diegomrsantos d46bcdb6ac
fix: compilation issue when `libp2p_expensive_metrics` is enabled. (#1014) 2024-01-29 11:31:11 +01:00
Jacek Sieka 2ede0fa40c
remove redundant gcsafe annotations (#999) 2023-12-05 08:05:32 +01:00
diegomrsantos 75871817ee
Split msgs in iwant response if bigger than limit (#944) 2023-10-02 11:39:28 +02:00
diegomrsantos 56599f5b9d
GossipSub Traffic scoring (#920) 2023-09-22 16:45:08 +02:00
Tanguy b784167805
GossipSub: IDontWant (#934) 2023-07-28 10:58:05 +02:00
Tanguy 66f9dc9167
Remove all `Result.get()`s & `Option` -> `Opt` (#902)
Co-authored-by: Ludovic Chenut <ludovic@status.im>
Co-authored-by: Diego <diego@status.im>
2023-06-28 16:44:58 +02:00
Tanguy 1c4d0832ce
Add GossipSub ping (#912) 2023-06-21 10:40:10 +02:00
Tanguy a65b7b028f
GossipSub: remove peer if we can't establish sendConn (#886) 2023-06-14 17:23:39 +02:00
Tanguy c76d1e18ef
Remove nim 1.2 support (#907) 2023-06-07 11:12:49 +00:00
Tanguy 4aa615c44c
GossipSub: TimedEntry & shortAgent fixes (#858) 2023-04-03 11:05:01 +02:00
Tanguy 6b61ce8c91
GossipSub: Better IWANT handling (#875) 2023-04-03 10:56:20 +02:00
Tanguy 8d5ea43e2b
Upgrade flow refactoring (#807) 2023-03-08 12:30:19 +01:00
Tanguy c11772c94e
Happy new year! (#847) 2023-01-20 15:47:40 +01:00
Tanguy 166c0d1c87
Fix PubSub subscribe on connection race condition (#809) 2023-01-10 13:33:14 +01:00
diegomrsantos a56c3bc296
Make observedAddr optional (#772)
Co-authored-by: Tanguy <tanguy@status.im>
2022-09-22 21:55:59 +02:00
Tanguy abbeaab684
Keep connection alive when peer doesn't support pubsub (#754) 2022-09-06 13:20:42 +02:00
Miran 2fbe82bf9d
make it more Nim 1.4+ compatible (#742) 2022-08-03 13:33:19 +02:00
tersec 1e598a0239
topicIDs -> topicIds; PubsubPeerEvent -> PubSubPeerEvent; messageIDs -> messageIds (#748) 2022-07-27 17:14:05 +00:00
tersec 4ca1c2d7ed
--styleCheck:error (#743)
* --styleCheck:error

* PeerID -> PeerId

* other libp2p styleCheck changes

* MessageID -> messageId; topicID -> topicId
2022-07-27 11:47:50 +00:00
Tanguy d0523fdc9d
Documentation (#716) 2022-07-01 20:19:57 +02:00
Tanguy df566e69db
Fixes for style check (#676) 2021-12-16 11:05:20 +01:00
Tanguy 5885e03680
Add maxMessageSize option to pubsub (#634)
* Add maxMessageSize option to pubsub
* Switched default to 1mb
2021-10-25 12:58:38 +02:00
Tanguy 846baf3853
Various cleanups part 1 (#632)
* raise -> raise exc
* replace stdlib random with bearssl
* object init -> new
* Remove deprecated procs
* getMandatoryField
2021-10-25 10:26:32 +02:00
Tanguy Cizain caac8191d2
Change newXXXX procs to XXXX.new (#585)
* newBufferStream -> BufferStream.new

* newMultistream -> MultistreamSelect.new

* newSecio -> Secio.new

* newNoise -> Noise.new

* newPlainText -> PlainText.new

* newPubSubPeer -> PubSubPeer.new

* newIdentify -> Identify.new

* newMuxerProvider -> MuxerProvider.new
2021-06-07 09:32:08 +02:00
Dmitriy Ryajov 3da656687b
use LPError more consistently (#582)
* use LPError more consistently

* don't use Exceptino

* annotate with raises

* don't panic on concatenation

* further rework error handling
2021-06-02 15:39:10 +02:00
Dmitriy Ryajov ac4e060e1a
adding raises defect across the codebase (#572)
* adding raises defect across the codebase

* use unittest2

* add windows deps caching

* update mingw link

* die on failed peerinfo initialization

* use result.expect instead of get

* use expect more consistently and rework inits

* use expect more consistently

* throw on missing public key

* remove unused closure annotation

* merge master
2021-05-21 10:27:01 -06:00
Jacek Sieka 83a20a992a
gossipsub: unsubscribe fixes (#569)
* gossipsub: unsubscribe fixes

* fix KeyError when updating metric of unsubscribed topic
* fix unsubscribe message not being sent to all peers causing them to
keep thinking we're still subscribed
* release memory earlier in a few places

* floodsub fix
2021-05-07 00:43:45 +02:00
Jacek Sieka e285d8bbf4
mem usage cleanups for pubsub (#564)
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.

In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.

To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)

* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
2021-04-18 10:08:33 +02:00
Jacek Sieka 70deac9e0d
fix peer score accumulation (#541)
* fix accumulating peer score
* fix missing exception handling
* remove unnecessary initHashSet/initTable calls
* simplify peer stats management
* clean up tests a little
* fix some missing raises annotations
2021-03-09 13:22:52 +01:00
Giovanni Petrantoni 02ad017107
Gossipsub fixes and Initiator flagging fixes (#539)
* properly propagate initiator information for gossipsub

* Fix pubsubpeer lifetime management

* restore old behavior

* tests fixing

* clamp backoff time value received

* fix member name collisions

* internal test fixes

* better names and explaining of the importance of transport direction

* fixes
2021-03-03 08:23:40 +09:00
Giovanni Petrantoni c1334c6d89 pubsubpeer better address management 2021-02-28 04:53:17 +00:00
Giovanni Petrantoni 7b2727d930 avoid leaking in peersInIP, don't depend on sendConn 2021-02-27 23:49:56 +09:00
Giovanni Petrantoni fae38e0146 fix PubSubPeer hashing issues 2021-02-26 19:19:15 +09:00
Giovanni Petrantoni fd73cf9f9d
Refactor gossipsub into multiple modules (#515)
* Refactor gossipsub into multiple modules

* splitup further gossipsub

* move more mesh related stuff to behavior

* fix internal tests

* fix PubSubPeer.outbound flag, make it more reliable

* use discard rather then _
2021-02-06 09:13:04 +09:00
Giovanni Petrantoni 240ec84ffb
Gossipsub wip (#502)
* Remove unused connections in pubsubpeer, also removed wrong usages, add a disconnect bad peers parameter

* handle exceptions in disconnectPeer

* small fix

* use the proper disconnection procedure for gossip peers

* fixes, more metrics add test about disconnection

* hot fix possible null pointers in switch

* silly isnil sugar

* Fix and test gossip directPeer connections
2021-01-15 13:48:03 +09:00
Giovanni Petrantoni b902c030a0
add metrics into chronosstream to identify peers agents (#458)
* add metrics into chronosstream to identify peers agents

* avoid too many agent strings

* use gauge instead of counter for stream metrics

* filter identity on /

* also track bytes traffic

* fix identity tracking closeimpl call

* add gossip rpc metrics

* fix missing metrics inclusions

* metrics fixes and additions

* add a KnownLibP2PAgents strdefine

* enforse toLowerAscii to agent names (metrics)

* incoming rpc metrics

* fix silly mistake in rpc metrics

* fix agent metrics logic

* libp2p_gossipsub_failed_publish metric

* message ids metrics

* libp2p_pubsub_broadcast_ihave metric improvement

* refactor expensive gossip metrics

* more detailed metrics

* metrics improvements

* remove generic metrics for `set` users

* small fixes, add debug counters

* fix counter and add missing subs metrics!

* agent metrics behind -d:libp2p_agents_metrics

* remove testing related code from this PR

* small rebroadcast metric fix

* fix small mistake

* add some guide to the readme in order to use new metrics

* add libp2p_gossipsub_peers_scores metric

* add protobuf metrics to understand bytes traffic precisely

* refactor gossipsub metrics

* remove unused variable

* add more metrics, refactor rebalance metrics

* avoid bad metric concurrent states

* use a stack structure for gossip mesh metrics

* refine sub metrics

* add received subs metrics fixes

* measure handlers of known topics

* sub/unsub counter

* unsubscribeAll log unknown topics

* expose a way to specify known topics at runtime
2021-01-08 14:21:24 +09:00
Dmitriy Ryajov d1c689e5ab
adding libp2p tag to logScope (#465) 2020-12-01 11:34:27 -06:00
Dmitriy Ryajov 034a1e8b1b
small cleanups from tcp-limits2 (#446) 2020-11-23 15:02:23 -06:00
Giovanni Petrantoni 4a98a8af5a
gossip pruning fixes related to #371 (#385)
* gossip pruning fixes related to #371

* better trace for grafted/pruned

* shorted azure testing again
2020-10-02 13:09:31 +09:00