83a20a992a
gossipsub: unsubscribe fixes ( #569 )
...
* gossipsub: unsubscribe fixes
* fix KeyError when updating metric of unsubscribed topic
* fix unsubscribe message not being sent to all peers causing them to
keep thinking we're still subscribed
* release memory earlier in a few places
* floodsub fix
2021-05-07 00:43:45 +02:00
9f301964ed
fix control messages ( #566 )
...
* remove unused control graft check in handleControl
* avoid sending empty Iwant messages
2021-04-28 10:03:03 +09:00
f81a085d0b
More gossip coverage ( #553 )
...
* add floodPublish test
* test delivery via control Iwant/have mechanics
* fix issues in control, and add testing
* fix possible backoff issue with pruned routine overriding it
2021-04-22 18:51:22 +09:00
e285d8bbf4
mem usage cleanups for pubsub ( #564 )
...
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.
In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.
To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)
* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
2021-04-18 10:08:33 +02:00
70deac9e0d
fix peer score accumulation ( #541 )
...
* fix accumulating peer score
* fix missing exception handling
* remove unnecessary initHashSet/initTable calls
* simplify peer stats management
* clean up tests a little
* fix some missing raises annotations
2021-03-09 13:22:52 +01:00
7b2727d930
avoid leaking in peersInIP, don't depend on sendConn
2021-02-27 23:49:56 +09:00
67d0926e89
use in any case PeerID for peersInIP to avoid keeping references
2021-02-27 21:31:59 +09:00
e124e342b0
n subscription limits ( #528 )
...
* subscription high water, cleanups
* subscription limits test
* newline
2021-02-12 12:27:26 +09:00
fff54fa23c
add more diagnostics when gossip publish fails
2021-02-09 18:42:59 +09:00
fd73cf9f9d
Refactor gossipsub into multiple modules ( #515 )
...
* Refactor gossipsub into multiple modules
* splitup further gossipsub
* move more mesh related stuff to behavior
* fix internal tests
* fix PubSubPeer.outbound flag, make it more reliable
* use discard rather then _
2021-02-06 09:13:04 +09:00
5aebf0990e
peer stats fixes ( #511 )
...
Gossipsub fix, required by nimbus, merging into master as low impact
2021-01-29 12:41:51 +09:00
1d77d37f17
Gossipsub scoring fixes ( #508 )
...
* Fix some problematics when running with full scoring
* more fixes
2021-01-25 21:13:42 +09:00
b57101f265
add an invalid topic subscriptions metric
2021-01-15 18:55:54 +09:00
1fb783eb7f
let apps decide if they want to penalize peers on invalid topic
2021-01-15 18:50:42 +09:00
6542b913df
set "ignoring invalid topic subscription" to trace level
2021-01-15 18:48:58 +09:00
240ec84ffb
Gossipsub wip ( #502 )
...
* Remove unused connections in pubsubpeer, also removed wrong usages, add a disconnect bad peers parameter
* handle exceptions in disconnectPeer
* small fix
* use the proper disconnection procedure for gossip peers
* fixes, more metrics add test about disconnection
* hot fix possible null pointers in switch
* silly isnil sugar
* Fix and test gossip directPeer connections
2021-01-15 13:48:03 +09:00
dc48170b0d
Gossip subscription improvements ( #497 )
...
* salt ids in seen table
* add subscription validation callback and avoid processing topics we don't care of
* apply penalty on bad subscription
* fix IHave handling IDs
* reduce indenting, add some comments
* fix gossip randombytes generation
* do not descore unwanted topics (might happen, due to timing, needs improvements)
* cleaning up and added tests
* validate subscriptions only when subscribing
* set notice level for failed publish
* fix floodsub behavior
2021-01-13 23:49:44 +09:00
b902c030a0
add metrics into chronosstream to identify peers agents ( #458 )
...
* add metrics into chronosstream to identify peers agents
* avoid too many agent strings
* use gauge instead of counter for stream metrics
* filter identity on /
* also track bytes traffic
* fix identity tracking closeimpl call
* add gossip rpc metrics
* fix missing metrics inclusions
* metrics fixes and additions
* add a KnownLibP2PAgents strdefine
* enforse toLowerAscii to agent names (metrics)
* incoming rpc metrics
* fix silly mistake in rpc metrics
* fix agent metrics logic
* libp2p_gossipsub_failed_publish metric
* message ids metrics
* libp2p_pubsub_broadcast_ihave metric improvement
* refactor expensive gossip metrics
* more detailed metrics
* metrics improvements
* remove generic metrics for `set` users
* small fixes, add debug counters
* fix counter and add missing subs metrics!
* agent metrics behind -d:libp2p_agents_metrics
* remove testing related code from this PR
* small rebroadcast metric fix
* fix small mistake
* add some guide to the readme in order to use new metrics
* add libp2p_gossipsub_peers_scores metric
* add protobuf metrics to understand bytes traffic precisely
* refactor gossipsub metrics
* remove unused variable
* add more metrics, refactor rebalance metrics
* avoid bad metric concurrent states
* use a stack structure for gossip mesh metrics
* refine sub metrics
* add received subs metrics fixes
* measure handlers of known topics
* sub/unsub counter
* unsubscribeAll log unknown topics
* expose a way to specify known topics at runtime
2021-01-08 14:21:24 +09:00
5e79d3ab9c
avoid triggering unsubscribeAll from unsubscribe (gossip)
2020-12-20 17:08:03 +09:00
4858e0ab15
Gossipsub refactor pt2 ( #495 )
...
* add sub/unsub test
* remove unused variable from gossip
2020-12-20 00:45:34 +09:00
05e789a34f
Gossipsub refactor ( #490 )
...
* refactor peerStats, re-enable scores for testing
* remove gossip 1.0
* cleanup
* codecov matrix fixes
* restore previous score on onNewPeer
* fix coverage n checks
* unsubscribeAll gossipsub fixes
* refactor unsub/sub
* refactor onNewPeer and fix score flow
* disable scores by default (change in tests later)
* fix tests, enable scores in tests
* fix wrongly merged test
* ensure topic removal from topics table
* small typo fix
* testinterop fixes
2020-12-19 15:43:32 +01:00
a1a5f9abac
nice msgid log formatting ( #491 )
2020-12-18 16:14:07 +01:00
52628d1a2e
avoid using debug logging in gossip, just because
2020-12-17 17:21:09 +09:00
9e5ba64c48
avoid creating prune message unless we're pruning ( #487 )
2020-12-15 22:46:03 +01:00
18d69a5c41
Workaround the gossipsub table race condition ( #486 )
2020-12-15 12:32:11 -06:00
f8f0bc1bd8
Gossip improvements ( #476 )
...
* add more traces, remove async from rebalance
* more traces
* avoid computng scores when weight is 0.0
* debug colocation, fix an indent in unsubpeer (minor)
* add full ValidationResult coverage
* store in cache only after validation
* gossip 1.0 fixes
* fix typo
* gossip 10 internal test fixes
* test fixing
* refactor peerstats usages
* populate tables if missing when scoring
2020-12-15 10:25:22 +09:00
d1c689e5ab
adding libp2p tag to logScope ( #465 )
2020-12-01 11:34:27 -06:00
e1648d4404
fix mcache logic check in gossipsub
2020-12-01 23:55:51 +09:00
b4738d723c
Some gossip fixes ( #467 )
...
* fix some missing rpc in rebalanceMesh
* clarify some variable names and lifetime
* further improvements
2020-12-01 11:44:09 +01:00
02b20440f2
Limit ihave emission ( #462 )
...
* add some limits to ihave emission, go has them, rust does not actually
* restore shuffling of IDs
* add some context
2020-11-28 16:27:39 +09:00
12db9a4cf2
TopicParams validation tuning
2020-11-28 00:23:09 +09:00
809df8d04d
add some extra gossip metrics
2020-11-26 16:20:34 +09:00
6c7f2766fe
expose seenTTL parameters ( #457 )
2020-11-26 14:45:10 +09:00
034a1e8b1b
small cleanups from tcp-limits2 ( #446 )
2020-11-23 15:02:23 -06:00
93b6c4dc52
Gossip runtime params ( #437 )
...
* move gossip parameters to runtime
* internal test fixes
* add missing params
* restore const parameters are soldi base and use them in init
* more constants tuning
2020-11-19 16:48:17 +09:00
a9948b0b05
clarify validation messages ( #431 )
...
* clarify validation messages
* add codecov threshold
2020-11-12 01:42:12 +09:00
e496802943
Least expensive metrics ( #421 )
...
* add more general and useful metrics
* fix gossipsub peers metrics in heartbeat
2020-11-04 15:18:00 +01:00
75b023c9e5
gossipsub audit fixes ( #412 )
...
* [SEC] gossipsub - rebalanceMesh grafts peers giving preference to low scores #405
* comment score choices
* compiler warning fixes/bug fixes (unsubscribe)
* rebalanceMesh does not enforce D_out quota
* fix outbound grafting
* fight the nim compiler
* fix closure capture bs...
* another closure fix
* #403 rebalance prune fixes
* more test fixing
* #403 fixes
* #402 avoid removing scores on unsub
* #401 handleGraft improvements
* [SEC] handleIHAVE/handleIWANT recommendations
* add a note about peer exchange handling
2020-10-30 21:49:54 +09:00
27b9bf436e
fix validation according to specification ( #410 )
2020-10-21 12:25:42 +09:00
556213abf4
Extended validators ( #395 )
...
* gossip extended validation
* fix flood tests
* fix gossip 1.0 tests
* synthax consistency
2020-10-12 16:56:00 +09:00
0f2435f551
better opportunistic grafting score (when score is disabled) ( #389 )
2020-10-03 09:26:45 +09:00
4a98a8af5a
gossip pruning fixes related to #371 ( #385 )
...
* gossip pruning fixes related to #371
* better trace for grafted/pruned
* shorted azure testing again
2020-10-02 13:09:31 +09:00
17e00e642a
limit write queue length ( #376 )
...
To break a potential read/write deadlock, gossipsub uses an unbounded
queue for writes - when peers are too slow to process this queue, it may
end up growing without bounds causing high memory usage.
Here, we introduce a maximum write queue length after which the peer is
disconnected - the queue is generous enough that any "normal" usage
should be fine - writes that are `await`:ed are not affected, only
writes that are launched in an `asyncSpawn` task or similar.
* avoid unnecessary copy of message when there are no send observers
* release message memory earlier in gossipsub
* simplify pubsubpeer logging
2020-09-24 18:43:20 +02:00
ec322124ac
allow to omit peerId and seqno ( #372 )
...
* allow to omit peerId and seqno
* small refactor
* wip
* fix message encoding
* improve rpc signature logic
* remove peerid from verify
* trace fixes
* fix message test
* fix gossip 1.0 tests
2020-09-23 17:56:33 +02:00
471e5906f6
fix gossipsub memory leak on disconnected peer ( #371 )
...
When messages can't be sent to peer, we try to establish a send
connection - this causes messages to stack up as more and more unsent
messages are blocked on the dial lock.
* remove dial lock
* run reconnection loop in background task
2020-09-22 09:05:53 +02:00
b99d2039a8
Gossip one one ( #240 )
...
* allow multiple codecs per protocol (without breaking things)
* add 1.1 protocol to gossip
* explicit peering part 1
* explicit peering part 2
* explicit peering part 3
* PeerInfo and ControlPrune protocols
* fix encodePrune
* validated always, even explicit peers
* prune by score (score is stub still)
* add a way to pass parameters to gossip
* standard setup fixes
* take into account explicit direct peers in publish
* add floodPublish logic
* small fixes, publish still half broken
* make sure to waitsub in sparse test
* use var semantics to optimize table access
* wip... lvalues don't work properly sadly...
* big publish refactor, replenish and balance
* fix internal tests
* use g.peers for fanout (todo: don't include flood peers)
* exclude non gossip from fanout
* internal test fixes
* fix flood tests
* fix test's trypublish
* test interop fixes
* make sure to not remove peers from gossip table
* restore old replenishFanout
* cleanups
* restore utility module import
* restore trace vs debug in gossip
* improve fanout replenish behavior further
* triage publish nil peers (issue is on master too but just hidden behind a if/in)
* getGossipPeers fixes
* remove topics from pubsubpeer (was unused)
* simplify rebalanceMesh (following spec) and make it finally reach D_high
* better diagnostics
* merge new pubsubpeer, copy 1.1 to new module
* fix up merge
* conditional enable gossip11 module
* add back topics in peers, re-enable flood publish
* add more heartbeat locking to prevent races
* actually lock the heartbeat
* minor fixes
* with sugar
* merge 1.0
* remove assertion in publish
* fix multistream 1.1 multi proto
* Fix merge oops
* wip
* fix gossip 11 upstream
* gossipsub11 -> gossipsub
* support interop testing
* tests fixing
* fix directchat build
* control prune updates (pb)
* wip parameters
* gossip internal tests fixes
* parameters wip
* finishup with params
* cleanups/wip
* small sugar
* grafted and pruned procs
* wip updateScores
* wip
* fix logging issue
* pubsubpeer, chronicles explicit override
* fix internal gossip tests
* wip
* tables troubleshooting
* score wip
* score wip
* fixes
* fix test utils generateNodes
* don't delete while iterating in score update
* fix grafted defect
* add a handleConnect in subscribeTopic
* pruning improvements
* wip
* score fixes
* post merge - builds gossip tests
* further merge fixes
* rebalance improvements and opportunistic grafting
* fix test for now
* restore explicit peering
* implement peer exchange graft message
* add an hard cap to PX
* backoff time management
* IWANT cap/budget
* Adaptive gossip dissemination
* outbound mesh quota, internal tests fixing
* oversub prune score based, finish outbound quota
* finishup with score and ihave budget
* use go daemon 0.3.0
* import fixes
* byScore cleanup score sorting
* remove pointless scaling in `/` Duration operator
* revert using libp2p org for daemon
* interop fixes
* fixes and cleanup
* remove heartbeat assertion, minor debug fixes
* logging improvements and cleaning up
* (to revert) add some traces
* add explicit topic to gossip rpcs
* pubsub merge fixes and type fix in switch
* Revert "(to revert) add some traces"
This reverts commit 4663eaab6cc336c81cee50bc54025cf0b7bcbd99.
* cleanup some now irrelevant todo
* shuffle peers anyway as score might be disabled
* add missing shuffle
* old merge fix
* more merge fixes
* debug improvements
* re-enable gossip internal tests
* add gossip10 fallback (dormant but tested)
* split gossipsub internal tests into 1.0 and 1.1
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-09-21 11:16:29 +02:00
c1856fda53
simplify and unify logging ( #353 )
...
* use short format for logging peerid
* log peerid:oid for connections
2020-09-06 10:31:47 +02:00
9b815efe8f
gossipsub: don't subscribe to floodsub also ( #352 )
2020-09-04 22:53:03 +02:00
6d91d61844
small cleanups & docs ( #347 )
...
* simplify gossipsub heartbeat start / stop
* avoid alloc in peerid check
* stop iterating over seq after unsubscribing item (could crash)
* don't crash on missing private key with enabled sigs (shouldn't happen
but...)
2020-09-04 18:31:43 +02:00
0b85192119
Remove asyncCheck from codebase. ( #345 )
...
* Remove asyncCheck from codebase.
* Replace all `discard` statements with new `asyncSpawn`.
* Bump `nim-chronos` requirement.
2020-09-04 18:30:45 +02:00
Jacek Sieka
Giovanni Petrantoni
Giovanni Petrantoni
Dmitriy Ryajov
Eugene Kabanov