dc48170b0d
Gossip subscription improvements ( #497 )
...
* salt ids in seen table
* add subscription validation callback and avoid processing topics we don't care of
* apply penalty on bad subscription
* fix IHave handling IDs
* reduce indenting, add some comments
* fix gossip randombytes generation
* do not descore unwanted topics (might happen, due to timing, needs improvements)
* cleaning up and added tests
* validate subscriptions only when subscribing
* set notice level for failed publish
* fix floodsub behavior
2021-01-13 23:49:44 +09:00
b902c030a0
add metrics into chronosstream to identify peers agents ( #458 )
...
* add metrics into chronosstream to identify peers agents
* avoid too many agent strings
* use gauge instead of counter for stream metrics
* filter identity on /
* also track bytes traffic
* fix identity tracking closeimpl call
* add gossip rpc metrics
* fix missing metrics inclusions
* metrics fixes and additions
* add a KnownLibP2PAgents strdefine
* enforse toLowerAscii to agent names (metrics)
* incoming rpc metrics
* fix silly mistake in rpc metrics
* fix agent metrics logic
* libp2p_gossipsub_failed_publish metric
* message ids metrics
* libp2p_pubsub_broadcast_ihave metric improvement
* refactor expensive gossip metrics
* more detailed metrics
* metrics improvements
* remove generic metrics for `set` users
* small fixes, add debug counters
* fix counter and add missing subs metrics!
* agent metrics behind -d:libp2p_agents_metrics
* remove testing related code from this PR
* small rebroadcast metric fix
* fix small mistake
* add some guide to the readme in order to use new metrics
* add libp2p_gossipsub_peers_scores metric
* add protobuf metrics to understand bytes traffic precisely
* refactor gossipsub metrics
* remove unused variable
* add more metrics, refactor rebalance metrics
* avoid bad metric concurrent states
* use a stack structure for gossip mesh metrics
* refine sub metrics
* add received subs metrics fixes
* measure handlers of known topics
* sub/unsub counter
* unsubscribeAll log unknown topics
* expose a way to specify known topics at runtime
2021-01-08 14:21:24 +09:00
b2ea5a3c77
Concurrent upgrades ( #489 )
...
* adding an upgraded event to conn
* set stopped flag asap
* trigger upgradded event on conn
* set concurrency limit for accepts
* backporting semaphore from tcp-limits2
* export unittests module
* make params explicit
* tone down debug logs
* adding semaphore tests
* use semaphore to throttle concurent upgrades
* add libp2p scope
* trigger upgraded event before any other events
* add event handler for connection upgrade
* cleanup upgraded event on conn close
* make upgrades slot release rebust
* dont forget to release slot on nil connection
* misc
* make sure semaphore is always released
* minor improvements and a nil check
* removing unneeded comment
* make upgradeMonitor a non-closure proc
* make sure the `upgraded` event is initialized
* handle exceptions in accepts when stopping
* don't leak exceptions when stopping accept loops
2021-01-04 12:59:05 -06:00
5e79d3ab9c
avoid triggering unsubscribeAll from unsubscribe (gossip)
2020-12-20 17:08:03 +09:00
4858e0ab15
Gossipsub refactor pt2 ( #495 )
...
* add sub/unsub test
* remove unused variable from gossip
2020-12-20 00:45:34 +09:00
05e789a34f
Gossipsub refactor ( #490 )
...
* refactor peerStats, re-enable scores for testing
* remove gossip 1.0
* cleanup
* codecov matrix fixes
* restore previous score on onNewPeer
* fix coverage n checks
* unsubscribeAll gossipsub fixes
* refactor unsub/sub
* refactor onNewPeer and fix score flow
* disable scores by default (change in tests later)
* fix tests, enable scores in tests
* fix wrongly merged test
* ensure topic removal from topics table
* small typo fix
* testinterop fixes
2020-12-19 15:43:32 +01:00
6c2e743ff3
Race condition in pubsub #469 ( #471 )
...
* Race condition in pubsub #469
* use allFinished
* improve cancellation handling
2020-12-19 00:56:46 +09:00
a1a5f9abac
nice msgid log formatting ( #491 )
2020-12-18 16:14:07 +01:00
52628d1a2e
avoid using debug logging in gossip, just because
2020-12-17 17:21:09 +09:00
0c331d5200
simplify noise frame construction ( #478 )
2020-12-16 13:10:06 +01:00
9e5ba64c48
avoid creating prune message unless we're pruning ( #487 )
2020-12-15 22:46:03 +01:00
18d69a5c41
Workaround the gossipsub table race condition ( #486 )
2020-12-15 12:32:11 -06:00
f8f0bc1bd8
Gossip improvements ( #476 )
...
* add more traces, remove async from rebalance
* more traces
* avoid computng scores when weight is 0.0
* debug colocation, fix an indent in unsubpeer (minor)
* add full ValidationResult coverage
* store in cache only after validation
* gossip 1.0 fixes
* fix typo
* gossip 10 internal test fixes
* test fixing
* refactor peerstats usages
* populate tables if missing when scoring
2020-12-15 10:25:22 +09:00
6f1ecc8df7
streamline socket read/write hot path ( #473 )
...
* streamline socket read/write hot path
This avoids some unnecessary memory copying on the hot path of noise /
mplex, as well as getting rid of a few futures - profiling shows that
this is one of the main culprits of small memory allocations, which
makes sense - this is where gossip fan-out happens.
* fewer futures (and corresponding closures) when sending lpchannel
messages
* avoid data copies when encrypting and framing noise messages
* avoid copying tuple when reading noise data (poor c codegen)
* fix setting eof flag in secure read
* write noise frames in one go
...and closing secure socket once is enough
2020-12-09 08:56:40 -06:00
1befeb8c2e
clean up peerid ( #470 )
...
* fix dangling cstring on error return
* remove some useless inlines
* less mallocs in shortlog
* proc -> func
* rename test
2020-12-03 13:53:16 -06:00
d1c689e5ab
adding libp2p tag to logScope ( #465 )
2020-12-01 11:34:27 -06:00
e1648d4404
fix mcache logic check in gossipsub
2020-12-01 23:55:51 +09:00
b4738d723c
Some gossip fixes ( #467 )
...
* fix some missing rpc in rebalanceMesh
* clarify some variable names and lifetime
* further improvements
2020-12-01 11:44:09 +01:00
18443dafc1
rework peer event to take an initiator flag ( #456 )
...
* rework peer event to take an initiator flag
* use correct direction for initiator
2020-11-28 10:59:47 -06:00
3d44fcb8b3
use cancelAndAwait to mitigate further hangs ( #459 )
2020-11-28 09:48:06 -06:00
02b20440f2
Limit ihave emission ( #462 )
...
* add some limits to ihave emission, go has them, rust does not actually
* restore shuffling of IDs
* add some context
2020-11-28 16:27:39 +09:00
12db9a4cf2
TopicParams validation tuning
2020-11-28 00:23:09 +09:00
809df8d04d
add some extra gossip metrics
2020-11-26 16:20:34 +09:00
6c7f2766fe
expose seenTTL parameters ( #457 )
2020-11-26 14:45:10 +09:00
7b1e652224
Allow custom identify agent string ( #451 )
...
* allow custom agent version string
* rework tests and add test for custom agent version
2020-11-25 07:42:02 -06:00
164892776b
get rid of hangs cleanup ( #453 )
2020-11-25 07:35:25 -06:00
21110636cb
fixing log level to avoid sacring users ( #452 )
2020-11-24 12:07:27 -06:00
69ae24dc8d
less leak prone cleanup ( #447 )
...
* less leak prone cleanup
* fix double allFinished
2020-11-23 18:22:15 -06:00
6cc3f4283a
update conn peerinfo instead of replacing ( #445 )
...
* update conn peerinfo instead of replacing
* remove unnecesary peerid var
2020-11-23 15:15:55 -06:00
034a1e8b1b
small cleanups from tcp-limits2 ( #446 )
2020-11-23 15:02:23 -06:00
93b6c4dc52
Gossip runtime params ( #437 )
...
* move gossip parameters to runtime
* internal test fixes
* add missing params
* restore const parameters are soldi base and use them in init
* more constants tuning
2020-11-19 16:48:17 +09:00
a9948b0b05
clarify validation messages ( #431 )
...
* clarify validation messages
* add codecov threshold
2020-11-12 01:42:12 +09:00
90921bff09
move some importance trace logs to debug ( #428 )
2020-11-09 22:14:46 -06:00
3956f3fd69
make sure all streams are tracked ( #422 )
...
* make sure all streams are tracked
* revert unnecesary change
2020-11-04 21:52:54 -06:00
e496802943
Least expensive metrics ( #421 )
...
* add more general and useful metrics
* fix gossipsub peers metrics in heartbeat
2020-11-04 15:18:00 +01:00
7b5259dbc7
Move triggers ( #416 )
...
* move event triggers to connmanager
* use base error type
* avoid deadlocks
* handle eof and closed when identifying incoming
* use `closeWait`
2020-11-02 14:35:26 -06:00
43a77e60a1
split stream counts by direction ( #418 )
2020-11-01 16:23:26 -06:00
03639f1446
Revert "Channel leaks ( #413 )" ( #417 )
...
This reverts commit 1de1d49223
2020-11-01 14:49:25 -06:00
75b023c9e5
gossipsub audit fixes ( #412 )
...
* [SEC] gossipsub - rebalanceMesh grafts peers giving preference to low scores #405
* comment score choices
* compiler warning fixes/bug fixes (unsubscribe)
* rebalanceMesh does not enforce D_out quota
* fix outbound grafting
* fight the nim compiler
* fix closure capture bs...
* another closure fix
* #403 rebalance prune fixes
* more test fixing
* #403 fixes
* #402 avoid removing scores on unsub
* #401 handleGraft improvements
* [SEC] handleIHAVE/handleIWANT recommendations
* add a note about peer exchange handling
2020-10-30 21:49:54 +09:00
1de1d49223
Channel leaks ( #413 )
...
* break stream tracking by type
* use closeWithEOF to await wrapped stream
* fix cancelation leaks
* fix channel leaks
* logging
* use close monitor and always call closeUnderlying
* don't use closeWithEOF
* removing close monitor
* logging
2020-10-27 11:21:03 -06:00
462da1f7a8
gossip MessageID as seq[byte] ( #391 )
...
* gossip MessageID as seq[byte]
* combina hashes in defaultMsgIdProvider
* wip
* fix defaultMsgIdProvider
2020-10-21 12:26:04 +09:00
27b9bf436e
fix validation according to specification ( #410 )
2020-10-21 12:25:42 +09:00
5c19668b2d
avoid verbose EOF messages in readOnce(secure) ( #411 )
...
* avoid verbose EOF messages in readOnce(secure)
* shorten azure tests further
2020-10-21 10:08:24 +09:00
32623b930e
handle secure errors in readOnce (secure) ( #397 )
...
* handle secure errors in readOnce(secure)
* small synthax fix
* fix mistake in readOnce's isNil
2020-10-19 14:13:14 +09:00
556213abf4
Extended validators ( #395 )
...
* gossip extended validation
* fix flood tests
* fix gossip 1.0 tests
* synthax consistency
2020-10-12 16:56:00 +09:00
e3bdb9eb13
decode properly ControlPrune ( #392 )
2020-10-09 09:12:38 +09:00
0f2435f551
better opportunistic grafting score (when score is disabled) ( #389 )
2020-10-03 09:26:45 +09:00
4a98a8af5a
gossip pruning fixes related to #371 ( #385 )
...
* gossip pruning fixes related to #371
* better trace for grafted/pruned
* shorted azure testing again
2020-10-02 13:09:31 +09:00
03f5bbba6d
saner logging ( #381 )
2020-09-29 09:40:06 -06:00
98d0cc3a16
defaultMsgIdProvider alternative/test anonymize ( #379 )
...
* defaultMsgIdProvider alternative/test anonymize
* avoid freeze during flood tests
* avoid `empty message, skipping` situation
* test observers
* avoid double initPubSub
* fix gossip testing (specially when anonymize is on)
* make azure tests shorter
2020-09-28 09:11:18 +02:00
Giovanni Petrantoni
Dmitriy Ryajov
Giovanni Petrantoni
Jacek Sieka
Mamy Ratsimbazafy