diegomrsantos
1a707e1264
feat: add max number of elements to non-prio queue ( #1077 )
2024-03-25 22:00:11 +01:00
Álex Cabeza Romero
458b0885dd
fix(issue-1052): Single topic for RPC Message ( #1061 )
2024-03-25 12:06:34 +01:00
Jacek Sieka
467b5b4f0c
avoid cancelling send future ( #1075 )
2024-03-20 10:54:32 +00:00
Jacek Sieka
ae13a0d583
Send priority with queue fix ( #1051 )
...
Co-authored-by: Diego <diego@status.im>
2024-03-05 15:05:21 +00:00
Diego
c5e4f8e12d
Revert "feat: message prioritization with immediate peer-published dispatch and queuing for other msgs ( #1015 )"
...
This reverts commit fe4ff798856f0fd2e9be75b61a7f461f5d8d3564.
2024-02-19 13:47:37 +01:00
diegomrsantos
fe4ff79885
feat: message prioritization with immediate peer-published dispatch and queuing for other msgs ( #1015 )
2024-02-16 10:54:16 +01:00
diegomrsantos
5594bcb33e
fix: more metrics issues when libp2p_expensive_metrics is enabled ( #1016 )
2024-01-30 16:55:55 +01:00
diegomrsantos
d46bcdb6ac
fix: compilation issue when libp2p_expensive_metrics is enabled. ( #1014 )
2024-01-29 11:31:11 +01:00
Jacek Sieka
2ede0fa40c
remove redundant gcsafe annotations ( #999 )
2023-12-05 08:05:32 +01:00
diegomrsantos
75871817ee
Split msgs in iwant response if bigger than limit ( #944 )
2023-10-02 11:39:28 +02:00
diegomrsantos
56599f5b9d
GossipSub Traffic scoring ( #920 )
2023-09-22 16:45:08 +02:00
Tanguy
b784167805
GossipSub: IDontWant ( #934 )
2023-07-28 10:58:05 +02:00
Tanguy
66f9dc9167
Remove all Result.get()s & Option -> Opt ( #902 )
...
Co-authored-by: Ludovic Chenut <ludovic@status.im>
Co-authored-by: Diego <diego@status.im>
2023-06-28 16:44:58 +02:00
Tanguy
1c4d0832ce
Add GossipSub ping ( #912 )
2023-06-21 10:40:10 +02:00
Tanguy
a65b7b028f
GossipSub: remove peer if we can't establish sendConn ( #886 )
2023-06-14 17:23:39 +02:00
Tanguy
c76d1e18ef
Remove nim 1.2 support ( #907 )
2023-06-07 11:12:49 +00:00
Tanguy
4aa615c44c
GossipSub: TimedEntry & shortAgent fixes ( #858 )
2023-04-03 11:05:01 +02:00
Tanguy
6b61ce8c91
GossipSub: Better IWANT handling ( #875 )
2023-04-03 10:56:20 +02:00
Tanguy
8d5ea43e2b
Upgrade flow refactoring ( #807 )
2023-03-08 12:30:19 +01:00
Tanguy
c11772c94e
Happy new year! ( #847 )
2023-01-20 15:47:40 +01:00
Tanguy
166c0d1c87
Fix PubSub subscribe on connection race condition ( #809 )
2023-01-10 13:33:14 +01:00
diegomrsantos
a56c3bc296
Make observedAddr optional ( #772 )
...
Co-authored-by: Tanguy <tanguy@status.im>
2022-09-22 21:55:59 +02:00
Tanguy
abbeaab684
Keep connection alive when peer doesn't support pubsub ( #754 )
2022-09-06 13:20:42 +02:00
Miran
2fbe82bf9d
make it more Nim 1.4+ compatible ( #742 )
2022-08-03 13:33:19 +02:00
tersec
1e598a0239
topicIDs -> topicIds; PubsubPeerEvent -> PubSubPeerEvent; messageIDs -> messageIds ( #748 )
2022-07-27 17:14:05 +00:00
tersec
4ca1c2d7ed
--styleCheck:error ( #743 )
...
* --styleCheck:error
* PeerID -> PeerId
* other libp2p styleCheck changes
* MessageID -> messageId; topicID -> topicId
2022-07-27 11:47:50 +00:00
Tanguy
d0523fdc9d
Documentation ( #716 )
2022-07-01 20:19:57 +02:00
Tanguy
df566e69db
Fixes for style check ( #676 )
2021-12-16 11:05:20 +01:00
Tanguy
5885e03680
Add maxMessageSize option to pubsub ( #634 )
...
* Add maxMessageSize option to pubsub
* Switched default to 1mb
2021-10-25 12:58:38 +02:00
Tanguy
846baf3853
Various cleanups part 1 ( #632 )
...
* raise -> raise exc
* replace stdlib random with bearssl
* object init -> new
* Remove deprecated procs
* getMandatoryField
2021-10-25 10:26:32 +02:00
Tanguy Cizain
caac8191d2
Change newXXXX procs to XXXX.new ( #585 )
...
* newBufferStream -> BufferStream.new
* newMultistream -> MultistreamSelect.new
* newSecio -> Secio.new
* newNoise -> Noise.new
* newPlainText -> PlainText.new
* newPubSubPeer -> PubSubPeer.new
* newIdentify -> Identify.new
* newMuxerProvider -> MuxerProvider.new
2021-06-07 09:32:08 +02:00
Dmitriy Ryajov
3da656687b
use LPError more consistently ( #582 )
...
* use LPError more consistently
* don't use Exceptino
* annotate with raises
* don't panic on concatenation
* further rework error handling
2021-06-02 15:39:10 +02:00
Dmitriy Ryajov
ac4e060e1a
adding raises defect across the codebase ( #572 )
...
* adding raises defect across the codebase
* use unittest2
* add windows deps caching
* update mingw link
* die on failed peerinfo initialization
* use result.expect instead of get
* use expect more consistently and rework inits
* use expect more consistently
* throw on missing public key
* remove unused closure annotation
* merge master
2021-05-21 10:27:01 -06:00
Jacek Sieka
83a20a992a
gossipsub: unsubscribe fixes ( #569 )
...
* gossipsub: unsubscribe fixes
* fix KeyError when updating metric of unsubscribed topic
* fix unsubscribe message not being sent to all peers causing them to
keep thinking we're still subscribed
* release memory earlier in a few places
* floodsub fix
2021-05-07 00:43:45 +02:00
Jacek Sieka
e285d8bbf4
mem usage cleanups for pubsub ( #564 )
...
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.
In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.
To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)
* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
2021-04-18 10:08:33 +02:00
Jacek Sieka
70deac9e0d
fix peer score accumulation ( #541 )
...
* fix accumulating peer score
* fix missing exception handling
* remove unnecessary initHashSet/initTable calls
* simplify peer stats management
* clean up tests a little
* fix some missing raises annotations
2021-03-09 13:22:52 +01:00
Giovanni Petrantoni
02ad017107
Gossipsub fixes and Initiator flagging fixes ( #539 )
...
* properly propagate initiator information for gossipsub
* Fix pubsubpeer lifetime management
* restore old behavior
* tests fixing
* clamp backoff time value received
* fix member name collisions
* internal test fixes
* better names and explaining of the importance of transport direction
* fixes
2021-03-03 08:23:40 +09:00
Giovanni Petrantoni
c1334c6d89
pubsubpeer better address management
2021-02-28 04:53:17 +00:00
Giovanni Petrantoni
7b2727d930
avoid leaking in peersInIP, don't depend on sendConn
2021-02-27 23:49:56 +09:00
Giovanni Petrantoni
fae38e0146
fix PubSubPeer hashing issues
2021-02-26 19:19:15 +09:00
Giovanni Petrantoni
fd73cf9f9d
Refactor gossipsub into multiple modules ( #515 )
...
* Refactor gossipsub into multiple modules
* splitup further gossipsub
* move more mesh related stuff to behavior
* fix internal tests
* fix PubSubPeer.outbound flag, make it more reliable
* use discard rather then _
2021-02-06 09:13:04 +09:00
Giovanni Petrantoni
240ec84ffb
Gossipsub wip ( #502 )
...
* Remove unused connections in pubsubpeer, also removed wrong usages, add a disconnect bad peers parameter
* handle exceptions in disconnectPeer
* small fix
* use the proper disconnection procedure for gossip peers
* fixes, more metrics add test about disconnection
* hot fix possible null pointers in switch
* silly isnil sugar
* Fix and test gossip directPeer connections
2021-01-15 13:48:03 +09:00
Giovanni Petrantoni
b902c030a0
add metrics into chronosstream to identify peers agents ( #458 )
...
* add metrics into chronosstream to identify peers agents
* avoid too many agent strings
* use gauge instead of counter for stream metrics
* filter identity on /
* also track bytes traffic
* fix identity tracking closeimpl call
* add gossip rpc metrics
* fix missing metrics inclusions
* metrics fixes and additions
* add a KnownLibP2PAgents strdefine
* enforse toLowerAscii to agent names (metrics)
* incoming rpc metrics
* fix silly mistake in rpc metrics
* fix agent metrics logic
* libp2p_gossipsub_failed_publish metric
* message ids metrics
* libp2p_pubsub_broadcast_ihave metric improvement
* refactor expensive gossip metrics
* more detailed metrics
* metrics improvements
* remove generic metrics for `set` users
* small fixes, add debug counters
* fix counter and add missing subs metrics!
* agent metrics behind -d:libp2p_agents_metrics
* remove testing related code from this PR
* small rebroadcast metric fix
* fix small mistake
* add some guide to the readme in order to use new metrics
* add libp2p_gossipsub_peers_scores metric
* add protobuf metrics to understand bytes traffic precisely
* refactor gossipsub metrics
* remove unused variable
* add more metrics, refactor rebalance metrics
* avoid bad metric concurrent states
* use a stack structure for gossip mesh metrics
* refine sub metrics
* add received subs metrics fixes
* measure handlers of known topics
* sub/unsub counter
* unsubscribeAll log unknown topics
* expose a way to specify known topics at runtime
2021-01-08 14:21:24 +09:00
Dmitriy Ryajov
d1c689e5ab
adding libp2p tag to logScope ( #465 )
2020-12-01 11:34:27 -06:00
Dmitriy Ryajov
034a1e8b1b
small cleanups from tcp-limits2 ( #446 )
2020-11-23 15:02:23 -06:00
Giovanni Petrantoni
4a98a8af5a
gossip pruning fixes related to #371 ( #385 )
...
* gossip pruning fixes related to #371
* better trace for grafted/pruned
* shorted azure testing again
2020-10-02 13:09:31 +09:00
Jacek Sieka
8ecef46738
reencode gossipsub messages with anonymization ( #378 )
...
This helps protect against clients sending more data than they should
and thus getting penalized on topics that require anonymity
2020-09-25 18:39:34 +02:00
Jacek Sieka
17e00e642a
limit write queue length ( #376 )
...
To break a potential read/write deadlock, gossipsub uses an unbounded
queue for writes - when peers are too slow to process this queue, it may
end up growing without bounds causing high memory usage.
Here, we introduce a maximum write queue length after which the peer is
disconnected - the queue is generous enough that any "normal" usage
should be fine - writes that are `await`:ed are not affected, only
writes that are launched in an `asyncSpawn` task or similar.
* avoid unnecessary copy of message when there are no send observers
* release message memory earlier in gossipsub
* simplify pubsubpeer logging
2020-09-24 18:43:20 +02:00
Jacek Sieka
471e5906f6
fix gossipsub memory leak on disconnected peer ( #371 )
...
When messages can't be sent to peer, we try to establish a send
connection - this causes messages to stack up as more and more unsent
messages are blocked on the dial lock.
* remove dial lock
* run reconnection loop in background task
2020-09-22 09:05:53 +02:00
Giovanni Petrantoni
b99d2039a8
Gossip one one ( #240 )
...
* allow multiple codecs per protocol (without breaking things)
* add 1.1 protocol to gossip
* explicit peering part 1
* explicit peering part 2
* explicit peering part 3
* PeerInfo and ControlPrune protocols
* fix encodePrune
* validated always, even explicit peers
* prune by score (score is stub still)
* add a way to pass parameters to gossip
* standard setup fixes
* take into account explicit direct peers in publish
* add floodPublish logic
* small fixes, publish still half broken
* make sure to waitsub in sparse test
* use var semantics to optimize table access
* wip... lvalues don't work properly sadly...
* big publish refactor, replenish and balance
* fix internal tests
* use g.peers for fanout (todo: don't include flood peers)
* exclude non gossip from fanout
* internal test fixes
* fix flood tests
* fix test's trypublish
* test interop fixes
* make sure to not remove peers from gossip table
* restore old replenishFanout
* cleanups
* restore utility module import
* restore trace vs debug in gossip
* improve fanout replenish behavior further
* triage publish nil peers (issue is on master too but just hidden behind a if/in)
* getGossipPeers fixes
* remove topics from pubsubpeer (was unused)
* simplify rebalanceMesh (following spec) and make it finally reach D_high
* better diagnostics
* merge new pubsubpeer, copy 1.1 to new module
* fix up merge
* conditional enable gossip11 module
* add back topics in peers, re-enable flood publish
* add more heartbeat locking to prevent races
* actually lock the heartbeat
* minor fixes
* with sugar
* merge 1.0
* remove assertion in publish
* fix multistream 1.1 multi proto
* Fix merge oops
* wip
* fix gossip 11 upstream
* gossipsub11 -> gossipsub
* support interop testing
* tests fixing
* fix directchat build
* control prune updates (pb)
* wip parameters
* gossip internal tests fixes
* parameters wip
* finishup with params
* cleanups/wip
* small sugar
* grafted and pruned procs
* wip updateScores
* wip
* fix logging issue
* pubsubpeer, chronicles explicit override
* fix internal gossip tests
* wip
* tables troubleshooting
* score wip
* score wip
* fixes
* fix test utils generateNodes
* don't delete while iterating in score update
* fix grafted defect
* add a handleConnect in subscribeTopic
* pruning improvements
* wip
* score fixes
* post merge - builds gossip tests
* further merge fixes
* rebalance improvements and opportunistic grafting
* fix test for now
* restore explicit peering
* implement peer exchange graft message
* add an hard cap to PX
* backoff time management
* IWANT cap/budget
* Adaptive gossip dissemination
* outbound mesh quota, internal tests fixing
* oversub prune score based, finish outbound quota
* finishup with score and ihave budget
* use go daemon 0.3.0
* import fixes
* byScore cleanup score sorting
* remove pointless scaling in `/` Duration operator
* revert using libp2p org for daemon
* interop fixes
* fixes and cleanup
* remove heartbeat assertion, minor debug fixes
* logging improvements and cleaning up
* (to revert) add some traces
* add explicit topic to gossip rpcs
* pubsub merge fixes and type fix in switch
* Revert "(to revert) add some traces"
This reverts commit 4663eaab6cc336c81cee50bc54025cf0b7bcbd99.
* cleanup some now irrelevant todo
* shuffle peers anyway as score might be disabled
* add missing shuffle
* old merge fix
* more merge fixes
* debug improvements
* re-enable gossip internal tests
* add gossip10 fallback (dormant but tested)
* split gossipsub internal tests into 1.0 and 1.1
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-09-21 11:16:29 +02:00
