3956f3fd69
make sure all streams are tracked ( #422 )
...
* make sure all streams are tracked
* revert unnecesary change
2020-11-04 21:52:54 -06:00
6040cb4ef1
fix debugutils ( #423 )
2020-11-04 19:56:28 -06:00
7cc42ce219
start adding more tests + minor fixes ( #419 )
...
* start adding more tests + minor fixes
* add wrong secure negotiation test
* add noise failed handshake test
2020-11-04 23:24:41 +09:00
e496802943
Least expensive metrics ( #421 )
...
* add more general and useful metrics
* fix gossipsub peers metrics in heartbeat
2020-11-04 15:18:00 +01:00
7b5259dbc7
Move triggers ( #416 )
...
* move event triggers to connmanager
* use base error type
* avoid deadlocks
* handle eof and closed when identifying incoming
* use `closeWait`
2020-11-02 14:35:26 -06:00
43a77e60a1
split stream counts by direction ( #418 )
2020-11-01 16:23:26 -06:00
03639f1446
Revert "Channel leaks ( #413 )" ( #417 )
...
This reverts commit 1de1d49223
2020-11-01 14:49:25 -06:00
9c1633bf87
fix ValidIpAddress multiaddress init return type
2020-10-31 13:20:29 +09:00
04c95cb7b0
Fix write should be writeArray.
2020-10-31 03:23:34 +02:00
ff48d0b1a2
Proper fix for init(ValidIpAddress).
2020-10-30 17:52:38 +02:00
3d9948a65e
ensure all multiaddress routines use Result
2020-10-30 23:50:04 +09:00
75b023c9e5
gossipsub audit fixes ( #412 )
...
* [SEC] gossipsub - rebalanceMesh grafts peers giving preference to low scores #405
* comment score choices
* compiler warning fixes/bug fixes (unsubscribe)
* rebalanceMesh does not enforce D_out quota
* fix outbound grafting
* fight the nim compiler
* fix closure capture bs...
* another closure fix
* #403 rebalance prune fixes
* more test fixing
* #403 fixes
* #402 avoid removing scores on unsub
* #401 handleGraft improvements
* [SEC] handleIHAVE/handleIWANT recommendations
* add a note about peer exchange handling
2020-10-30 21:49:54 +09:00
1de1d49223
Channel leaks ( #413 )
...
* break stream tracking by type
* use closeWithEOF to await wrapped stream
* fix cancelation leaks
* fix channel leaks
* logging
* use close monitor and always call closeUnderlying
* don't use closeWithEOF
* removing close monitor
* logging
2020-10-27 11:21:03 -06:00
eeaa62feec
add more debug details to multiaddress assertions
2020-10-21 18:29:59 +09:00
462da1f7a8
gossip MessageID as seq[byte] ( #391 )
...
* gossip MessageID as seq[byte]
* combina hashes in defaultMsgIdProvider
* wip
* fix defaultMsgIdProvider
2020-10-21 12:26:04 +09:00
27b9bf436e
fix validation according to specification ( #410 )
2020-10-21 12:25:42 +09:00
5c19668b2d
avoid verbose EOF messages in readOnce(secure) ( #411 )
...
* avoid verbose EOF messages in readOnce(secure)
* shorten azure tests further
2020-10-21 10:08:24 +09:00
9c58356823
change repository name for nimbus auto-bump
2020-10-19 14:39:05 +09:00
32623b930e
handle secure errors in readOnce (secure) ( #397 )
...
* handle secure errors in readOnce(secure)
* small synthax fix
* fix mistake in readOnce's isNil
2020-10-19 14:13:14 +09:00
bd70515087
fix first interop test ( #398 )
2020-10-15 10:37:20 +09:00
556213abf4
Extended validators ( #395 )
...
* gossip extended validation
* fix flood tests
* fix gossip 1.0 tests
* synthax consistency
2020-10-12 16:56:00 +09:00
c81b665b0d
replace nim-beacon-chain URLs with nimbus-eth2 URLs ( #396 )
2020-10-09 18:02:03 +02:00
e3bdb9eb13
decode properly ControlPrune ( #392 )
2020-10-09 09:12:38 +09:00
98d82fce5c
fix opportunistic graft in internal 11 testing ( #390 )
2020-10-05 11:35:03 +09:00
0f2435f551
better opportunistic grafting score (when score is disabled) ( #389 )
2020-10-03 09:26:45 +09:00
853238a215
feature/expose-matcher ( #387 )
...
* exposes matcher
* might work
* fix
* fix
2020-10-02 08:59:15 -06:00
4a98a8af5a
gossip pruning fixes related to #371 ( #385 )
...
* gossip pruning fixes related to #371
* better trace for grafted/pruned
* shorted azure testing again
2020-10-02 13:09:31 +09:00
03f5bbba6d
saner logging ( #381 )
2020-09-29 09:40:06 -06:00
98d0cc3a16
defaultMsgIdProvider alternative/test anonymize ( #379 )
...
* defaultMsgIdProvider alternative/test anonymize
* avoid freeze during flood tests
* avoid `empty message, skipping` situation
* test observers
* avoid double initPubSub
* fix gossip testing (specially when anonymize is on)
* make azure tests shorter
2020-09-28 09:11:18 +02:00
8ecef46738
reencode gossipsub messages with anonymization ( #378 )
...
This helps protect against clients sending more data than they should
and thus getting penalized on topics that require anonymity
2020-09-25 18:39:34 +02:00
17e00e642a
limit write queue length ( #376 )
...
To break a potential read/write deadlock, gossipsub uses an unbounded
queue for writes - when peers are too slow to process this queue, it may
end up growing without bounds causing high memory usage.
Here, we introduce a maximum write queue length after which the peer is
disconnected - the queue is generous enough that any "normal" usage
should be fine - writes that are `await`:ed are not affected, only
writes that are launched in an `asyncSpawn` task or similar.
* avoid unnecessary copy of message when there are no send observers
* release message memory earlier in gossipsub
* simplify pubsubpeer logging
2020-09-24 18:43:20 +02:00
58f26d4164
temporarily skip broken test on windows ( #375 )
2020-09-24 14:50:04 +02:00
25bd0a18f4
small fixes ( #374 )
...
* add helper to read EOF marker after closing stream (else stream stay
alive until timeout/reset)
* don't assert on empty channel message
* don't loop when writing to chronos (no need)
2020-09-24 07:30:19 +02:00
ec322124ac
allow to omit peerId and seqno ( #372 )
...
* allow to omit peerId and seqno
* small refactor
* wip
* fix message encoding
* improve rpc signature logic
* remove peerid from verify
* trace fixes
* fix message test
* fix gossip 1.0 tests
2020-09-23 17:56:33 +02:00
abd234601b
move events to conn manager ( #373 )
2020-09-23 08:07:16 -06:00
471e5906f6
fix gossipsub memory leak on disconnected peer ( #371 )
...
When messages can't be sent to peer, we try to establish a send
connection - this causes messages to stack up as more and more unsent
messages are blocked on the dial lock.
* remove dial lock
* run reconnection loop in background task
2020-09-22 09:05:53 +02:00
49a12e619d
channel close race and deadlock fixes ( #368 )
...
* channel close race and deadlock fixes
* remove send lock, write chunks in one go
* push some of half-closed implementation to BufferStream
* fix some hangs where LPChannel readers and writers would not always
wake up
* simplify lazy channels
* fix close happening more than once in some orderings
* reenable connection tracking tests
* close channels first on mplex close such that consumers can read bytes
A notable difference is that BufferedStream is no longer considered EOF
until someone has actually read the EOF marker.
* docs, simplification
2020-09-21 19:48:19 +02:00
b99d2039a8
Gossip one one ( #240 )
...
* allow multiple codecs per protocol (without breaking things)
* add 1.1 protocol to gossip
* explicit peering part 1
* explicit peering part 2
* explicit peering part 3
* PeerInfo and ControlPrune protocols
* fix encodePrune
* validated always, even explicit peers
* prune by score (score is stub still)
* add a way to pass parameters to gossip
* standard setup fixes
* take into account explicit direct peers in publish
* add floodPublish logic
* small fixes, publish still half broken
* make sure to waitsub in sparse test
* use var semantics to optimize table access
* wip... lvalues don't work properly sadly...
* big publish refactor, replenish and balance
* fix internal tests
* use g.peers for fanout (todo: don't include flood peers)
* exclude non gossip from fanout
* internal test fixes
* fix flood tests
* fix test's trypublish
* test interop fixes
* make sure to not remove peers from gossip table
* restore old replenishFanout
* cleanups
* restore utility module import
* restore trace vs debug in gossip
* improve fanout replenish behavior further
* triage publish nil peers (issue is on master too but just hidden behind a if/in)
* getGossipPeers fixes
* remove topics from pubsubpeer (was unused)
* simplify rebalanceMesh (following spec) and make it finally reach D_high
* better diagnostics
* merge new pubsubpeer, copy 1.1 to new module
* fix up merge
* conditional enable gossip11 module
* add back topics in peers, re-enable flood publish
* add more heartbeat locking to prevent races
* actually lock the heartbeat
* minor fixes
* with sugar
* merge 1.0
* remove assertion in publish
* fix multistream 1.1 multi proto
* Fix merge oops
* wip
* fix gossip 11 upstream
* gossipsub11 -> gossipsub
* support interop testing
* tests fixing
* fix directchat build
* control prune updates (pb)
* wip parameters
* gossip internal tests fixes
* parameters wip
* finishup with params
* cleanups/wip
* small sugar
* grafted and pruned procs
* wip updateScores
* wip
* fix logging issue
* pubsubpeer, chronicles explicit override
* fix internal gossip tests
* wip
* tables troubleshooting
* score wip
* score wip
* fixes
* fix test utils generateNodes
* don't delete while iterating in score update
* fix grafted defect
* add a handleConnect in subscribeTopic
* pruning improvements
* wip
* score fixes
* post merge - builds gossip tests
* further merge fixes
* rebalance improvements and opportunistic grafting
* fix test for now
* restore explicit peering
* implement peer exchange graft message
* add an hard cap to PX
* backoff time management
* IWANT cap/budget
* Adaptive gossip dissemination
* outbound mesh quota, internal tests fixing
* oversub prune score based, finish outbound quota
* finishup with score and ihave budget
* use go daemon 0.3.0
* import fixes
* byScore cleanup score sorting
* remove pointless scaling in `/` Duration operator
* revert using libp2p org for daemon
* interop fixes
* fixes and cleanup
* remove heartbeat assertion, minor debug fixes
* logging improvements and cleaning up
* (to revert) add some traces
* add explicit topic to gossip rpcs
* pubsub merge fixes and type fix in switch
* Revert "(to revert) add some traces"
This reverts commit 4663eaab6cc336c81cee50bc54025cf0b7bcbd99.
* cleanup some now irrelevant todo
* shuffle peers anyway as score might be disabled
* add missing shuffle
* old merge fix
* more merge fixes
* debug improvements
* re-enable gossip internal tests
* add gossip10 fallback (dormant but tested)
* split gossipsub internal tests into 1.0 and 1.1
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-09-21 11:16:29 +02:00
b7e5d1122c
cleanups ( #366 )
...
* reuse connection timeout for noise handshake (avoid extra timer)
* enforce nbytes > 0 for readOnce
* avoid some unnecessary memory zeroing
* simplify noise
* fix dumping when noise splits message
2020-09-16 11:55:25 +02:00
b0d86b95dd
add peer lifecycle events ( #357 )
...
* add peer lifecycle events
* rework peer events to not use connection events
* don't use result in pubsub and switch init
* wip
* use ordered hashes and remove logscope
* logging
* add missing test
* small fixes
2020-09-15 14:19:22 -06:00
a6007be428
avoid sending empty seqno and/or fromPeer (gossip rpc) ( #364 )
2020-09-15 12:33:18 +02:00
e2d46c6b53
Add `libp2p_dump` and `libp2p_dump_dir` compiler time options. ( #359 )
...
* Add `libp2p_dump` and `libp2p_dump_dir` compiler time options.
* Add tools/pbcap_parser.
* Add mplex control messages decoding.
2020-09-15 11:16:43 +02:00
5e66f6fbd8
Add logScope to connmanager and pubsubprotobuf ( #363 )
2020-09-15 08:03:53 +02:00
0db45462cd
mplex fixes ( #362 )
...
* remove almost-empty types module
* lock when writing message (that's the only place the lock matters, and
only when the message is > max msg size)
* logging updates (log in consistent order, makes reading logs easier)
* raise EOF from readExactly only if no bytes have been read (to signal
that _no_ bytes were lost)
2020-09-14 10:19:54 +02:00
96d4c44fec
refactor bufferstream to use a queue ( #346 )
...
This change modifies how the backpressure algorithm in bufferstream
works - in particular, instead of working byte-by-byte, it will now work
seq-by-seq.
When data arrives, it usually does so in packets - in the current
bufferstream, the packet is read then split into bytes which are fed one
by one to the bufferstream. On the reading side, the bytes are popped of
the bufferstream, again byte by byte, to satisfy `readOnce` requests -
this introduces a lot of synchronization traffic because the checks for
full buffer and for async event handling must be done for every byte.
In this PR, a queue of length 1 is used instead - this means there will
at most exist one "packet" in `pushTo`, one in the queue and one in the
slush buffer that is used to store incomplete reads.
* avoid byte-by-byte copy to buffer, with synchronization in-between
* reuse AsyncQueue synchronization logic instead of rolling own
* avoid writeHandler callback - implement `write` method instead
* simplify EOF signalling by only setting EOF flag in queue reader (and
reset)
* remove BufferStream pipes (unused)
* fixes drainBuffer deadlock when drain is called from within read loop
and thus blocks draining
* fix lpchannel init order
2020-09-10 08:19:13 +02:00
5b347adf58
logging fixes and small cleanups ( #361 )
...
In particular, allow longer multistream select reads
2020-09-09 19:12:08 +02:00
63b38734bd
fix poor performance in LRU cache ( #360 )
...
it turns out (in NBC) a heap is sufficiently slow becuase of all the
deletes that it makes more sense to go with a linked list
2020-09-09 18:28:46 +02:00
82c179db9e
mplex fixes ( #356 )
...
* close the right connection when channel send fails
* don't crash on channel id that is not unique
2020-09-08 08:24:28 +02:00
2b72d485a3
a few more log fixes ( #355 )
2020-09-07 14:15:11 +02:00
c1856fda53
simplify and unify logging ( #353 )
...
* use short format for logging peerid
* log peerid:oid for connections
2020-09-06 10:31:47 +02:00
Dmitriy Ryajov
Giovanni Petrantoni
Jacek Sieka
Giovanni Petrantoni
cheatfate
Giovanni Petrantoni
tersec
Dean Eigenmann
Mamy Ratsimbazafy
Oskar Thorén