Commit Graph

129 Commits

Author SHA1 Message Date
Álex 48846d69cb
chore(logs): remove duplicate msg key (#1180)
* Remove `msg` parameter key in logs.

closes: https://github.com/vacp2p/nim-libp2p/issues/1176
2024-08-14 17:19:54 +02:00
gabrielmer a60f0c5532
feat: adding onValidated observer (#1128)
### Description

Adding an `onValidated` observer which will run every time a message is
received and validated. This comes from the necessity of precisely track
message deliveries and network activity.

`onRecv` observers run before any check is performed on the received
message, which means that it runs every time a duplicate or invalid
message arrives, which is inefficient and inaccurate for our purpose of
tracking only received, unique and valid messages. Therefore, adding
this extra option of running an observer for every message after all
validation checks pass.
2024-08-01 18:50:13 +03:00
diegomrsantos 96bfefc928
feat(gossipsub): support version 1.2.0 (#1106) 2024-06-12 15:46:47 +02:00
diegomrsantos dc83a1e9b6
chore(formatting): format the whole codebase using nph 0.5.1 (#1118) 2024-06-11 17:18:06 +02:00
diegomrsantos 88e233db81
fix: Asynchronous task [sendMsgSlow()] was cancelled [FutureDefect] (#1094) 2024-05-07 15:44:14 +02:00
Jacek Sieka 84659af45b
avoid latency/copy when sending low-priority messages to fast peers (#1060) 2024-05-02 12:26:16 +02:00
Jacek Sieka aef44ed1ce
salt `idontwant` (#1090) 2024-05-02 12:18:55 +02:00
diegomrsantos 1a707e1264
feat: add max number of elements to non-prio queue (#1077) 2024-03-25 22:00:11 +01:00
Álex Cabeza Romero 458b0885dd
fix(issue-1052): Single topic for RPC Message (#1061) 2024-03-25 12:06:34 +01:00
Jacek Sieka 467b5b4f0c
avoid cancelling send future (#1075) 2024-03-20 10:54:32 +00:00
Jacek Sieka ae13a0d583
Send priority with queue fix (#1051)
Co-authored-by: Diego <diego@status.im>
2024-03-05 15:05:21 +00:00
Diego c5e4f8e12d
Revert "feat: message prioritization with immediate peer-published dispatch and queuing for other msgs (#1015)"
This reverts commit fe4ff79885.
2024-02-19 13:47:37 +01:00
diegomrsantos fe4ff79885
feat: message prioritization with immediate peer-published dispatch and queuing for other msgs (#1015) 2024-02-16 10:54:16 +01:00
diegomrsantos 5594bcb33e
fix: more metrics issues when libp2p_expensive_metrics is enabled (#1016) 2024-01-30 16:55:55 +01:00
diegomrsantos d46bcdb6ac
fix: compilation issue when `libp2p_expensive_metrics` is enabled. (#1014) 2024-01-29 11:31:11 +01:00
Jacek Sieka 2ede0fa40c
remove redundant gcsafe annotations (#999) 2023-12-05 08:05:32 +01:00
diegomrsantos 75871817ee
Split msgs in iwant response if bigger than limit (#944) 2023-10-02 11:39:28 +02:00
diegomrsantos 56599f5b9d
GossipSub Traffic scoring (#920) 2023-09-22 16:45:08 +02:00
Tanguy b784167805
GossipSub: IDontWant (#934) 2023-07-28 10:58:05 +02:00
Tanguy 66f9dc9167
Remove all `Result.get()`s & `Option` -> `Opt` (#902)
Co-authored-by: Ludovic Chenut <ludovic@status.im>
Co-authored-by: Diego <diego@status.im>
2023-06-28 16:44:58 +02:00
Tanguy 1c4d0832ce
Add GossipSub ping (#912) 2023-06-21 10:40:10 +02:00
Tanguy a65b7b028f
GossipSub: remove peer if we can't establish sendConn (#886) 2023-06-14 17:23:39 +02:00
Tanguy c76d1e18ef
Remove nim 1.2 support (#907) 2023-06-07 11:12:49 +00:00
Tanguy 4aa615c44c
GossipSub: TimedEntry & shortAgent fixes (#858) 2023-04-03 11:05:01 +02:00
Tanguy 6b61ce8c91
GossipSub: Better IWANT handling (#875) 2023-04-03 10:56:20 +02:00
Tanguy 8d5ea43e2b
Upgrade flow refactoring (#807) 2023-03-08 12:30:19 +01:00
Tanguy c11772c94e
Happy new year! (#847) 2023-01-20 15:47:40 +01:00
Tanguy 166c0d1c87
Fix PubSub subscribe on connection race condition (#809) 2023-01-10 13:33:14 +01:00
diegomrsantos a56c3bc296
Make observedAddr optional (#772)
Co-authored-by: Tanguy <tanguy@status.im>
2022-09-22 21:55:59 +02:00
Tanguy abbeaab684
Keep connection alive when peer doesn't support pubsub (#754) 2022-09-06 13:20:42 +02:00
Miran 2fbe82bf9d
make it more Nim 1.4+ compatible (#742) 2022-08-03 13:33:19 +02:00
tersec 1e598a0239
topicIDs -> topicIds; PubsubPeerEvent -> PubSubPeerEvent; messageIDs -> messageIds (#748) 2022-07-27 17:14:05 +00:00
tersec 4ca1c2d7ed
--styleCheck:error (#743)
* --styleCheck:error

* PeerID -> PeerId

* other libp2p styleCheck changes

* MessageID -> messageId; topicID -> topicId
2022-07-27 11:47:50 +00:00
Tanguy d0523fdc9d
Documentation (#716) 2022-07-01 20:19:57 +02:00
Tanguy df566e69db
Fixes for style check (#676) 2021-12-16 11:05:20 +01:00
Tanguy 5885e03680
Add maxMessageSize option to pubsub (#634)
* Add maxMessageSize option to pubsub
* Switched default to 1mb
2021-10-25 12:58:38 +02:00
Tanguy 846baf3853
Various cleanups part 1 (#632)
* raise -> raise exc
* replace stdlib random with bearssl
* object init -> new
* Remove deprecated procs
* getMandatoryField
2021-10-25 10:26:32 +02:00
Tanguy Cizain caac8191d2
Change newXXXX procs to XXXX.new (#585)
* newBufferStream -> BufferStream.new

* newMultistream -> MultistreamSelect.new

* newSecio -> Secio.new

* newNoise -> Noise.new

* newPlainText -> PlainText.new

* newPubSubPeer -> PubSubPeer.new

* newIdentify -> Identify.new

* newMuxerProvider -> MuxerProvider.new
2021-06-07 09:32:08 +02:00
Dmitriy Ryajov 3da656687b
use LPError more consistently (#582)
* use LPError more consistently

* don't use Exceptino

* annotate with raises

* don't panic on concatenation

* further rework error handling
2021-06-02 15:39:10 +02:00
Dmitriy Ryajov ac4e060e1a
adding raises defect across the codebase (#572)
* adding raises defect across the codebase

* use unittest2

* add windows deps caching

* update mingw link

* die on failed peerinfo initialization

* use result.expect instead of get

* use expect more consistently and rework inits

* use expect more consistently

* throw on missing public key

* remove unused closure annotation

* merge master
2021-05-21 10:27:01 -06:00
Jacek Sieka 83a20a992a
gossipsub: unsubscribe fixes (#569)
* gossipsub: unsubscribe fixes

* fix KeyError when updating metric of unsubscribed topic
* fix unsubscribe message not being sent to all peers causing them to
keep thinking we're still subscribed
* release memory earlier in a few places

* floodsub fix
2021-05-07 00:43:45 +02:00
Jacek Sieka e285d8bbf4
mem usage cleanups for pubsub (#564)
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.

In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.

To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)

* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
2021-04-18 10:08:33 +02:00
Jacek Sieka 70deac9e0d
fix peer score accumulation (#541)
* fix accumulating peer score
* fix missing exception handling
* remove unnecessary initHashSet/initTable calls
* simplify peer stats management
* clean up tests a little
* fix some missing raises annotations
2021-03-09 13:22:52 +01:00
Giovanni Petrantoni 02ad017107
Gossipsub fixes and Initiator flagging fixes (#539)
* properly propagate initiator information for gossipsub

* Fix pubsubpeer lifetime management

* restore old behavior

* tests fixing

* clamp backoff time value received

* fix member name collisions

* internal test fixes

* better names and explaining of the importance of transport direction

* fixes
2021-03-03 08:23:40 +09:00
Giovanni Petrantoni c1334c6d89 pubsubpeer better address management 2021-02-28 04:53:17 +00:00
Giovanni Petrantoni 7b2727d930 avoid leaking in peersInIP, don't depend on sendConn 2021-02-27 23:49:56 +09:00
Giovanni Petrantoni fae38e0146 fix PubSubPeer hashing issues 2021-02-26 19:19:15 +09:00
Giovanni Petrantoni fd73cf9f9d
Refactor gossipsub into multiple modules (#515)
* Refactor gossipsub into multiple modules

* splitup further gossipsub

* move more mesh related stuff to behavior

* fix internal tests

* fix PubSubPeer.outbound flag, make it more reliable

* use discard rather then _
2021-02-06 09:13:04 +09:00
Giovanni Petrantoni 240ec84ffb
Gossipsub wip (#502)
* Remove unused connections in pubsubpeer, also removed wrong usages, add a disconnect bad peers parameter

* handle exceptions in disconnectPeer

* small fix

* use the proper disconnection procedure for gossip peers

* fixes, more metrics add test about disconnection

* hot fix possible null pointers in switch

* silly isnil sugar

* Fix and test gossip directPeer connections
2021-01-15 13:48:03 +09:00
Giovanni Petrantoni b902c030a0
add metrics into chronosstream to identify peers agents (#458)
* add metrics into chronosstream to identify peers agents

* avoid too many agent strings

* use gauge instead of counter for stream metrics

* filter identity on /

* also track bytes traffic

* fix identity tracking closeimpl call

* add gossip rpc metrics

* fix missing metrics inclusions

* metrics fixes and additions

* add a KnownLibP2PAgents strdefine

* enforse toLowerAscii to agent names (metrics)

* incoming rpc metrics

* fix silly mistake in rpc metrics

* fix agent metrics logic

* libp2p_gossipsub_failed_publish metric

* message ids metrics

* libp2p_pubsub_broadcast_ihave metric improvement

* refactor expensive gossip metrics

* more detailed metrics

* metrics improvements

* remove generic metrics for `set` users

* small fixes, add debug counters

* fix counter and add missing subs metrics!

* agent metrics behind -d:libp2p_agents_metrics

* remove testing related code from this PR

* small rebroadcast metric fix

* fix small mistake

* add some guide to the readme in order to use new metrics

* add libp2p_gossipsub_peers_scores metric

* add protobuf metrics to understand bytes traffic precisely

* refactor gossipsub metrics

* remove unused variable

* add more metrics, refactor rebalance metrics

* avoid bad metric concurrent states

* use a stack structure for gossip mesh metrics

* refine sub metrics

* add received subs metrics fixes

* measure handlers of known topics

* sub/unsub counter

* unsubscribeAll log unknown topics

* expose a way to specify known topics at runtime
2021-01-08 14:21:24 +09:00