dc48170b0d
Gossip subscription improvements ( #497 )
...
* salt ids in seen table
* add subscription validation callback and avoid processing topics we don't care of
* apply penalty on bad subscription
* fix IHave handling IDs
* reduce indenting, add some comments
* fix gossip randombytes generation
* do not descore unwanted topics (might happen, due to timing, needs improvements)
* cleaning up and added tests
* validate subscriptions only when subscribing
* set notice level for failed publish
* fix floodsub behavior
2021-01-13 23:49:44 +09:00
b902c030a0
add metrics into chronosstream to identify peers agents ( #458 )
...
* add metrics into chronosstream to identify peers agents
* avoid too many agent strings
* use gauge instead of counter for stream metrics
* filter identity on /
* also track bytes traffic
* fix identity tracking closeimpl call
* add gossip rpc metrics
* fix missing metrics inclusions
* metrics fixes and additions
* add a KnownLibP2PAgents strdefine
* enforse toLowerAscii to agent names (metrics)
* incoming rpc metrics
* fix silly mistake in rpc metrics
* fix agent metrics logic
* libp2p_gossipsub_failed_publish metric
* message ids metrics
* libp2p_pubsub_broadcast_ihave metric improvement
* refactor expensive gossip metrics
* more detailed metrics
* metrics improvements
* remove generic metrics for `set` users
* small fixes, add debug counters
* fix counter and add missing subs metrics!
* agent metrics behind -d:libp2p_agents_metrics
* remove testing related code from this PR
* small rebroadcast metric fix
* fix small mistake
* add some guide to the readme in order to use new metrics
* add libp2p_gossipsub_peers_scores metric
* add protobuf metrics to understand bytes traffic precisely
* refactor gossipsub metrics
* remove unused variable
* add more metrics, refactor rebalance metrics
* avoid bad metric concurrent states
* use a stack structure for gossip mesh metrics
* refine sub metrics
* add received subs metrics fixes
* measure handlers of known topics
* sub/unsub counter
* unsubscribeAll log unknown topics
* expose a way to specify known topics at runtime
2021-01-08 14:21:24 +09:00
5e79d3ab9c
avoid triggering unsubscribeAll from unsubscribe (gossip)
2020-12-20 17:08:03 +09:00
4858e0ab15
Gossipsub refactor pt2 ( #495 )
...
* add sub/unsub test
* remove unused variable from gossip
2020-12-20 00:45:34 +09:00
05e789a34f
Gossipsub refactor ( #490 )
...
* refactor peerStats, re-enable scores for testing
* remove gossip 1.0
* cleanup
* codecov matrix fixes
* restore previous score on onNewPeer
* fix coverage n checks
* unsubscribeAll gossipsub fixes
* refactor unsub/sub
* refactor onNewPeer and fix score flow
* disable scores by default (change in tests later)
* fix tests, enable scores in tests
* fix wrongly merged test
* ensure topic removal from topics table
* small typo fix
* testinterop fixes
2020-12-19 15:43:32 +01:00
6c2e743ff3
Race condition in pubsub #469 ( #471 )
...
* Race condition in pubsub #469
* use allFinished
* improve cancellation handling
2020-12-19 00:56:46 +09:00
a1a5f9abac
nice msgid log formatting ( #491 )
2020-12-18 16:14:07 +01:00
52628d1a2e
avoid using debug logging in gossip, just because
2020-12-17 17:21:09 +09:00
9e5ba64c48
avoid creating prune message unless we're pruning ( #487 )
2020-12-15 22:46:03 +01:00
18d69a5c41
Workaround the gossipsub table race condition ( #486 )
2020-12-15 12:32:11 -06:00
f8f0bc1bd8
Gossip improvements ( #476 )
...
* add more traces, remove async from rebalance
* more traces
* avoid computng scores when weight is 0.0
* debug colocation, fix an indent in unsubpeer (minor)
* add full ValidationResult coverage
* store in cache only after validation
* gossip 1.0 fixes
* fix typo
* gossip 10 internal test fixes
* test fixing
* refactor peerstats usages
* populate tables if missing when scoring
2020-12-15 10:25:22 +09:00
1befeb8c2e
clean up peerid ( #470 )
...
* fix dangling cstring on error return
* remove some useless inlines
* less mallocs in shortlog
* proc -> func
* rename test
2020-12-03 13:53:16 -06:00
d1c689e5ab
adding libp2p tag to logScope ( #465 )
2020-12-01 11:34:27 -06:00
e1648d4404
fix mcache logic check in gossipsub
2020-12-01 23:55:51 +09:00
b4738d723c
Some gossip fixes ( #467 )
...
* fix some missing rpc in rebalanceMesh
* clarify some variable names and lifetime
* further improvements
2020-12-01 11:44:09 +01:00
18443dafc1
rework peer event to take an initiator flag ( #456 )
...
* rework peer event to take an initiator flag
* use correct direction for initiator
2020-11-28 10:59:47 -06:00
02b20440f2
Limit ihave emission ( #462 )
...
* add some limits to ihave emission, go has them, rust does not actually
* restore shuffling of IDs
* add some context
2020-11-28 16:27:39 +09:00
12db9a4cf2
TopicParams validation tuning
2020-11-28 00:23:09 +09:00
809df8d04d
add some extra gossip metrics
2020-11-26 16:20:34 +09:00
6c7f2766fe
expose seenTTL parameters ( #457 )
2020-11-26 14:45:10 +09:00
034a1e8b1b
small cleanups from tcp-limits2 ( #446 )
2020-11-23 15:02:23 -06:00
93b6c4dc52
Gossip runtime params ( #437 )
...
* move gossip parameters to runtime
* internal test fixes
* add missing params
* restore const parameters are soldi base and use them in init
* more constants tuning
2020-11-19 16:48:17 +09:00
a9948b0b05
clarify validation messages ( #431 )
...
* clarify validation messages
* add codecov threshold
2020-11-12 01:42:12 +09:00
90921bff09
move some importance trace logs to debug ( #428 )
2020-11-09 22:14:46 -06:00
e496802943
Least expensive metrics ( #421 )
...
* add more general and useful metrics
* fix gossipsub peers metrics in heartbeat
2020-11-04 15:18:00 +01:00
03639f1446
Revert "Channel leaks ( #413 )" ( #417 )
...
This reverts commit 1de1d49223
2020-11-01 14:49:25 -06:00
75b023c9e5
gossipsub audit fixes ( #412 )
...
* [SEC] gossipsub - rebalanceMesh grafts peers giving preference to low scores #405
* comment score choices
* compiler warning fixes/bug fixes (unsubscribe)
* rebalanceMesh does not enforce D_out quota
* fix outbound grafting
* fight the nim compiler
* fix closure capture bs...
* another closure fix
* #403 rebalance prune fixes
* more test fixing
* #403 fixes
* #402 avoid removing scores on unsub
* #401 handleGraft improvements
* [SEC] handleIHAVE/handleIWANT recommendations
* add a note about peer exchange handling
2020-10-30 21:49:54 +09:00
1de1d49223
Channel leaks ( #413 )
...
* break stream tracking by type
* use closeWithEOF to await wrapped stream
* fix cancelation leaks
* fix channel leaks
* logging
* use close monitor and always call closeUnderlying
* don't use closeWithEOF
* removing close monitor
* logging
2020-10-27 11:21:03 -06:00
462da1f7a8
gossip MessageID as seq[byte] ( #391 )
...
* gossip MessageID as seq[byte]
* combina hashes in defaultMsgIdProvider
* wip
* fix defaultMsgIdProvider
2020-10-21 12:26:04 +09:00
27b9bf436e
fix validation according to specification ( #410 )
2020-10-21 12:25:42 +09:00
556213abf4
Extended validators ( #395 )
...
* gossip extended validation
* fix flood tests
* fix gossip 1.0 tests
* synthax consistency
2020-10-12 16:56:00 +09:00
e3bdb9eb13
decode properly ControlPrune ( #392 )
2020-10-09 09:12:38 +09:00
0f2435f551
better opportunistic grafting score (when score is disabled) ( #389 )
2020-10-03 09:26:45 +09:00
4a98a8af5a
gossip pruning fixes related to #371 ( #385 )
...
* gossip pruning fixes related to #371
* better trace for grafted/pruned
* shorted azure testing again
2020-10-02 13:09:31 +09:00
03f5bbba6d
saner logging ( #381 )
2020-09-29 09:40:06 -06:00
98d0cc3a16
defaultMsgIdProvider alternative/test anonymize ( #379 )
...
* defaultMsgIdProvider alternative/test anonymize
* avoid freeze during flood tests
* avoid `empty message, skipping` situation
* test observers
* avoid double initPubSub
* fix gossip testing (specially when anonymize is on)
* make azure tests shorter
2020-09-28 09:11:18 +02:00
8ecef46738
reencode gossipsub messages with anonymization ( #378 )
...
This helps protect against clients sending more data than they should
and thus getting penalized on topics that require anonymity
2020-09-25 18:39:34 +02:00
17e00e642a
limit write queue length ( #376 )
...
To break a potential read/write deadlock, gossipsub uses an unbounded
queue for writes - when peers are too slow to process this queue, it may
end up growing without bounds causing high memory usage.
Here, we introduce a maximum write queue length after which the peer is
disconnected - the queue is generous enough that any "normal" usage
should be fine - writes that are `await`:ed are not affected, only
writes that are launched in an `asyncSpawn` task or similar.
* avoid unnecessary copy of message when there are no send observers
* release message memory earlier in gossipsub
* simplify pubsubpeer logging
2020-09-24 18:43:20 +02:00
25bd0a18f4
small fixes ( #374 )
...
* add helper to read EOF marker after closing stream (else stream stay
alive until timeout/reset)
* don't assert on empty channel message
* don't loop when writing to chronos (no need)
2020-09-24 07:30:19 +02:00
ec322124ac
allow to omit peerId and seqno ( #372 )
...
* allow to omit peerId and seqno
* small refactor
* wip
* fix message encoding
* improve rpc signature logic
* remove peerid from verify
* trace fixes
* fix message test
* fix gossip 1.0 tests
2020-09-23 17:56:33 +02:00
471e5906f6
fix gossipsub memory leak on disconnected peer ( #371 )
...
When messages can't be sent to peer, we try to establish a send
connection - this causes messages to stack up as more and more unsent
messages are blocked on the dial lock.
* remove dial lock
* run reconnection loop in background task
2020-09-22 09:05:53 +02:00
b99d2039a8
Gossip one one ( #240 )
...
* allow multiple codecs per protocol (without breaking things)
* add 1.1 protocol to gossip
* explicit peering part 1
* explicit peering part 2
* explicit peering part 3
* PeerInfo and ControlPrune protocols
* fix encodePrune
* validated always, even explicit peers
* prune by score (score is stub still)
* add a way to pass parameters to gossip
* standard setup fixes
* take into account explicit direct peers in publish
* add floodPublish logic
* small fixes, publish still half broken
* make sure to waitsub in sparse test
* use var semantics to optimize table access
* wip... lvalues don't work properly sadly...
* big publish refactor, replenish and balance
* fix internal tests
* use g.peers for fanout (todo: don't include flood peers)
* exclude non gossip from fanout
* internal test fixes
* fix flood tests
* fix test's trypublish
* test interop fixes
* make sure to not remove peers from gossip table
* restore old replenishFanout
* cleanups
* restore utility module import
* restore trace vs debug in gossip
* improve fanout replenish behavior further
* triage publish nil peers (issue is on master too but just hidden behind a if/in)
* getGossipPeers fixes
* remove topics from pubsubpeer (was unused)
* simplify rebalanceMesh (following spec) and make it finally reach D_high
* better diagnostics
* merge new pubsubpeer, copy 1.1 to new module
* fix up merge
* conditional enable gossip11 module
* add back topics in peers, re-enable flood publish
* add more heartbeat locking to prevent races
* actually lock the heartbeat
* minor fixes
* with sugar
* merge 1.0
* remove assertion in publish
* fix multistream 1.1 multi proto
* Fix merge oops
* wip
* fix gossip 11 upstream
* gossipsub11 -> gossipsub
* support interop testing
* tests fixing
* fix directchat build
* control prune updates (pb)
* wip parameters
* gossip internal tests fixes
* parameters wip
* finishup with params
* cleanups/wip
* small sugar
* grafted and pruned procs
* wip updateScores
* wip
* fix logging issue
* pubsubpeer, chronicles explicit override
* fix internal gossip tests
* wip
* tables troubleshooting
* score wip
* score wip
* fixes
* fix test utils generateNodes
* don't delete while iterating in score update
* fix grafted defect
* add a handleConnect in subscribeTopic
* pruning improvements
* wip
* score fixes
* post merge - builds gossip tests
* further merge fixes
* rebalance improvements and opportunistic grafting
* fix test for now
* restore explicit peering
* implement peer exchange graft message
* add an hard cap to PX
* backoff time management
* IWANT cap/budget
* Adaptive gossip dissemination
* outbound mesh quota, internal tests fixing
* oversub prune score based, finish outbound quota
* finishup with score and ihave budget
* use go daemon 0.3.0
* import fixes
* byScore cleanup score sorting
* remove pointless scaling in `/` Duration operator
* revert using libp2p org for daemon
* interop fixes
* fixes and cleanup
* remove heartbeat assertion, minor debug fixes
* logging improvements and cleaning up
* (to revert) add some traces
* add explicit topic to gossip rpcs
* pubsub merge fixes and type fix in switch
* Revert "(to revert) add some traces"
This reverts commit 4663eaab6c
2020-09-21 11:16:29 +02:00
b0d86b95dd
add peer lifecycle events ( #357 )
...
* add peer lifecycle events
* rework peer events to not use connection events
* don't use result in pubsub and switch init
* wip
* use ordered hashes and remove logscope
* logging
* add missing test
* small fixes
2020-09-15 14:19:22 -06:00
a6007be428
avoid sending empty seqno and/or fromPeer (gossip rpc) ( #364 )
2020-09-15 12:33:18 +02:00
5e66f6fbd8
Add logScope to connmanager and pubsubprotobuf ( #363 )
2020-09-15 08:03:53 +02:00
96d4c44fec
refactor bufferstream to use a queue ( #346 )
...
This change modifies how the backpressure algorithm in bufferstream
works - in particular, instead of working byte-by-byte, it will now work
seq-by-seq.
When data arrives, it usually does so in packets - in the current
bufferstream, the packet is read then split into bytes which are fed one
by one to the bufferstream. On the reading side, the bytes are popped of
the bufferstream, again byte by byte, to satisfy `readOnce` requests -
this introduces a lot of synchronization traffic because the checks for
full buffer and for async event handling must be done for every byte.
In this PR, a queue of length 1 is used instead - this means there will
at most exist one "packet" in `pushTo`, one in the queue and one in the
slush buffer that is used to store incomplete reads.
* avoid byte-by-byte copy to buffer, with synchronization in-between
* reuse AsyncQueue synchronization logic instead of rolling own
* avoid writeHandler callback - implement `write` method instead
* simplify EOF signalling by only setting EOF flag in queue reader (and
reset)
* remove BufferStream pipes (unused)
* fixes drainBuffer deadlock when drain is called from within read loop
and thus blocks draining
* fix lpchannel init order
2020-09-10 08:19:13 +02:00
5b347adf58
logging fixes and small cleanups ( #361 )
...
In particular, allow longer multistream select reads
2020-09-09 19:12:08 +02:00
63b38734bd
fix poor performance in LRU cache ( #360 )
...
it turns out (in NBC) a heap is sufficiently slow becuase of all the
deletes that it makes more sense to go with a linked list
2020-09-09 18:28:46 +02:00
c1856fda53
simplify and unify logging ( #353 )
...
* use short format for logging peerid
* log peerid:oid for connections
2020-09-06 10:31:47 +02:00
9b815efe8f
gossipsub: don't subscribe to floodsub also ( #352 )
2020-09-04 22:53:03 +02:00
Giovanni Petrantoni
Giovanni Petrantoni
Jacek Sieka
Dmitriy Ryajov
Mamy Ratsimbazafy
Oskar Thorén