Commit Graph

84 Commits

Author SHA1 Message Date
Dmitriy Ryajov e9d4679059
Race in connection setup (#464)
* check that connection is not closed or eof

* don't release connection lock prematurely

* test that only valid connections can be added

* correct exception type on closed connection

* add clarifying comment

* use closeWithEOF for more stable test

* misc comments

* log stream id in buffestream asserts

* use closeWithEOF to prevent races in tests

* give some time to the remote handler to trigger

* adding more tests to make codecov happy
2020-12-02 19:24:48 -06:00
Dmitriy Ryajov 1d16d22f5f
Don't allow concurrent pushdata (#444)
* handle resets properly with/without pushes/reads

* add clarifying comments

* pushEof should also not be concurrent

* move channel reset to bufferstream

this is where the action happens - lpchannel merely redefines how close
is done

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-11-23 09:07:11 -06:00
Dmitriy Ryajov 92fa4110c1
Rework transport to use chronos accept (#420)
* rework transport to use the new accept api

* use the new chronos primits

* fixup tests to use the new transport api

* handle all exceptions in upgradeIncoming

* master merge

* add multiaddress exception type

* raise appropriate exception on invalida address

* allow retrying on TransportTooManyError

* adding TODO

* wip

* merge master

* add sleep if nil is returned

* accept loop handles all exceptions

* avoid issues with tray/except/finally

* make consistent with master

* cleanup accept loop

* logging

* Update libp2p/transports/tcptransport.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* use Direction enum instead of initiator flag

* use consistent import style

* remove experimental `closeWithEOF()`

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-11-18 20:06:42 -06:00
Jacek Sieka 74acd0a33a
fix channels not being reset (#439)
* fix channels not being reset

silly for loop..

* allow only one concurrent read
* fix mplex test race condition
* add some bufferstream eof tests

* deadlock, lost data and hung channel fixes

* prevent concurrent `reset` calls
* reset LPChannel when read is cancelled (since data is lost)
* ensure there's one, and one only, 0-byte readOnce on EOF
* ensure that all data is returned before EOF is returned
* keep running activity monitor for half-closed channels (or they never
get closed)
2020-11-17 08:59:25 -06:00
Jacek Sieka 51a0aec058
read failure (#436)
* read failure

Test showing read failure on cancel

* adding one more test case from jacek

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-11-13 12:30:14 -06:00
Jacek Sieka 5e6ec6f422
Revert "failing test"
This reverts commit 5dbed426c2.
2020-11-13 12:26:08 +01:00
Jacek Sieka 5dbed426c2
failing test 2020-11-13 12:24:23 +01:00
Dmitriy Ryajov 55b763264e
Cleanup tests (#435)
* add async testing methods

* refactor with async testing methods

* use iffy in async tests
2020-11-12 21:44:02 -06:00
Dmitriy Ryajov 331961ef14
dont use double allFinished in mplex tests (#427) 2020-11-06 12:20:59 -06:00
Dmitriy Ryajov 4fb3f50d2c
Reset channels on close (#425)
* reset when failed to read/write muxed conn

* add more comprehensive resource cleanup tests

* style

* cleanup tests
2020-11-06 09:24:24 -06:00
Jacek Sieka 49a12e619d
channel close race and deadlock fixes (#368)
* channel close race and deadlock fixes

* remove send lock, write chunks in one go
* push some of half-closed implementation to BufferStream
* fix some hangs where LPChannel readers and writers would not always
wake up
* simplify lazy channels
* fix close happening more than once in some orderings
* reenable connection tracking tests
* close channels first on mplex close such that consumers can read bytes

A notable difference is that BufferedStream is no longer considered EOF
until someone has actually read the EOF marker.

* docs, simplification
2020-09-21 19:48:19 +02:00
Jacek Sieka 0db45462cd
mplex fixes (#362)
* remove almost-empty types module
* lock when writing message (that's the only place the lock matters, and
only when the message is > max msg size)
* logging updates (log in consistent order, makes reading logs easier)
* raise EOF from readExactly only if no bytes have been read (to signal
that _no_ bytes were lost)
2020-09-14 10:19:54 +02:00
Jacek Sieka 96d4c44fec
refactor bufferstream to use a queue (#346)
This change modifies how the backpressure algorithm in bufferstream
works - in particular, instead of working byte-by-byte, it will now work
seq-by-seq.

When data arrives, it usually does so in packets - in the current
bufferstream, the packet is read then split into bytes which are fed one
by one to the bufferstream. On the reading side, the bytes are popped of
the bufferstream, again byte by byte, to satisfy `readOnce` requests -
this introduces a lot of synchronization traffic because the checks for
full buffer and for async event handling must be done for every byte.

In this PR, a queue of length 1 is used instead - this means there will
at most exist one "packet" in `pushTo`, one in the queue and one in the
slush buffer that is used to store incomplete reads.

* avoid byte-by-byte copy to buffer, with synchronization in-between
* reuse AsyncQueue synchronization logic instead of rolling own
* avoid writeHandler callback - implement `write` method instead
* simplify EOF signalling by only setting EOF flag in queue reader (and
reset)
* remove BufferStream pipes (unused)
* fixes drainBuffer deadlock when drain is called from within read loop
and thus blocks draining
* fix lpchannel init order
2020-09-10 08:19:13 +02:00
Jacek Sieka ab864fc747
logging cleanups and small fixes (#331) 2020-08-15 21:50:31 +02:00
Dmitriy Ryajov d1f1e1b31e
add missing mplex half closed test (#326) 2020-08-12 07:23:49 +02:00
Dmitriy Ryajov 2325692f55
Fix half closed (#324)
* don't call `close` in `remoteClose`

* make sure timeout are properly propagted

* fix tests

* adding remote close write test
2020-08-10 16:17:11 -06:00
Giovanni Petrantoni 5c986cf657
Fix build, add some raises (#315)
* Fix build, add some raises

* wip

* wip more raises

* missing exc object in mplex

* proper lifetime for subscribePeer

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-08-05 19:30:57 -06:00
Dmitriy Ryajov 94196fee71
Connections and pubsub peers cleanup (#279)
* better peer tracking and cleanup

* check if peer and conn is nil

* test name

* make timeout more agressive

* rename method for better clarity
2020-07-17 13:46:24 -06:00
Dmitriy Ryajov ba071cafa6
Channel timeout (#278)
* add support for channel timeouts

* tests for channel timeout

* add timeouts to standard switch

* fix mplex init

* cleanup timer on stream close

* add comment for `isConnected`

* move cleanup event
2020-07-17 12:44:41 -06:00
Dmitriy Ryajov c788a6a3c0
Cleanup resources (#246)
* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises
2020-06-29 09:15:31 -06:00
Dmitriy Ryajov 5b28e8c488
Cleanup lpstream, Connection and BufferStream (#228)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* don't use delegation in connection

* move connection out to own file

* don't breakout on reset

* make sure to call close on secured conn

* add lpstream tracing

* don't breackdown by conn id

* fix import

* remove unused lable

* reset  connection on exception

* add additional metrics for skipped messages

* check for nil in secure.close
2020-06-19 11:29:43 -06:00
Dmitriy Ryajov bb8bff2195
add sparse message propagation tests to gossipsub (#202)
* add sparce tests to gossipsub

* add send hooks

* remove `all`
2020-06-02 17:53:38 -06:00
Dmitriy Ryajov 20c68a2018 use all() for futures and track connections 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 8e48bb117d rebasing master 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 77017d1e7e inc count before closing 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 8963c2d0ed fixes after backporting 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 46daed9a38 wip 2020-06-02 09:10:27 -06:00
Giovanni Petrantoni 6affcda937
Less exceptions more results (#188)
* Less exceptions more results

* Fix daemonapi and interop tests

* Add multibase

* wip multiaddress

* fix the build, consuming new result types

* fix standard setup

* Simplify match, rename into MaError, add more exaustive err text

* Fix the CI issues

* Fix directchat build

* daemon api fixes

* better err messages formatting

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-05-31 16:22:49 +02:00
Dmitriy Ryajov 7900fd9f61
Half closed (#174)
* call write until all is written out

* add comments to lpchannel fields

* add an eof flag to signal which end closed

* wip: rework with proper half-closed

* add eof and closed handling

* propagate closes to piped

* call parent close

* moving bufferstream trackers out

* move writeLock to bufferstream

* move writeLock out

* remove unused call

* wip

* rebasing master

* fix mplex tests

* wip

* fix bufferstream after backport

* wip

* rename to differentiate from chronos tracker

* close connection on chronos close

* make reset request asyncCheck

* fix channel cleanup

* misc

* don't use read

* fix backports

* make noise work again

* proper exception handling

* don't reraise just yet

* add convenience templates

* dont double wrap

* use async pragma

* fixes after backporting

* muxer owns connection

* remove on transport close cleanup

* revert back allread

* adding some todos

* read from stream

* inc count before closing

* rebasing master

* rebase master

* use correct exception type

* use try/finally insted of defer

* fix compile in trace mode

* reset channels on mplex close
2020-05-19 18:14:15 -06:00
Dmitriy Ryajov 9cf1fd0216
remove generic constructor and expose serverflags (#176)
* remove generic constructor and expose serverflags

* fix transport constructor

* fix merge issues
2020-05-18 13:04:05 -06:00
Dmitriy Ryajov 773b738c12
don't track Connection, track StreamTransport (#177)
* don't track Connection, track StreamTransport

* make tests more deterministic
2020-05-18 11:05:34 -06:00
Dmitriy Ryajov 1819502fb5
Cleanup - tests and logging (#178)
* make async for proper exception handling

* tryAndWarn msg messes up Exception msg

* misc: comment out tracker dumps

* cleanup mplex tests

* more informative errors

* give CI time to run

* revert change, bacause it causes races
2020-05-18 07:49:49 -06:00
Dmitriy Ryajov 167f42ed45
Remove read (#171)
* use readExactly

* remove `read`

* remove read

* no more `read`
2020-05-14 22:02:05 -06:00
Jacek Sieka 3053f03814 fix varint issues
* fixes #111
2020-05-11 09:12:23 -06:00
Jacek Sieka ccd019b328
use stream directly in chronosstream (#163)
* use stream directly in chronosstream

for now, chronos.AsyncStream is not used to provide any features on top
of chronos.Stream, so in order to simplify the code, chronosstream can
be used directly.

In particular, the exception handling is broken in the current
chronosstream - opening and closing the stream is simplified this way as
well.

A future implementation that actually takes advantage of the AsyncStream
features would wrap AsyncStream instead as a separate lpstream
implementation, leaving this one as-is.

* work around chronos exception type issue
2020-05-08 22:10:06 +02:00
Jacek Sieka 330da51819
removals (#159)
* remove unused stream methods
* reimplement some of them with proc's
* remove broken tests
* Error->Defect for defect
* warning fixes
2020-05-06 18:31:47 +02:00
Giovanni Petrantoni 4c6a123d31
Add chronos trackers and used them to sanitize resource disposal (#131)
* Add chronos trackers and used them to sanitize resource disposal

* Chronos trackers for transport tests wip

* No more chronos leaks in testtransport

* Make tcp transport and test more robust when closing

* Test async leaking tracking wip

* Fix a regression in wire connect

* Add chronos trackers to more tests and sanitize resource closure

* Wip fixing floodsub tests

* Floodsub wip

* Made floodsub basically deterministic, hit a nim bug with captures tho

* Wrap up floodsub tests refactor

* Wrapping up

* Add allFuturesThrowing utility

* Fix missing allFuturesThrowing in noise tests!

* Make tests green

* attempt fixing gossipsub failing cases

* Make sure to check also fanout in waitSub

* More verbose traces

* Gossipsub test improvments

* Refactor TcpTransport remove asyncCheck

* Add Connection trackers

* Add stricter connection tracking, wip mplex fix

* More asynccheck removal, in order to avoid connection leaks

* bump chronicles requirement

* Enable tracker dump to check CI output

* Wait for more futures in testmplex

* Remove tracker dump messages

* add tryAndWarn utility, fix mplex issue with go interop

* All allFuturesThrowing to directchat too

* make sure to cleanup on transport close
2020-04-21 10:24:42 +09:00
Giovanni Petrantoni e39bf0a4cf
Mplex: Add the ability to send any size payload (#123)
* Mplex: Add the ability to send any size payload

* Ensure size of coder header
2020-04-03 17:26:46 +02:00
Dmitriy Ryajov 6bb4e91a39 fix tests 2020-03-28 09:29:43 -06:00
Dmitriy Ryajov 3effb95f10 close underlying bufferstream in lpchannel 2020-03-28 09:29:43 -06:00
Dmitriy Ryajov 5c234f704d added 1 byte jitter test 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov dc97595aad remove irrelevant comment 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov 59abd2e4cd remove unnecesary while 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov 4fc84cbe81 add jitter tolerance to mplex 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov 8c96de5397 reduce waiting time from 5 seconds to 500 millis in msg size test 2020-03-10 12:43:32 -06:00
Eugene Kabanov 381630f185
Fix and refactoring of some procedures which are able to return nil as result (#97)
* Fix do not return nil as result.

* Fix mplex test to properly raise.
2020-03-04 21:45:14 +02:00
Dmitriy Ryajov 30a496de5e add TODO to explain why half closed tests failed 2020-02-16 11:31:35 -06:00
Dmitriy Ryajov 7f8eb0272e cleanup and fix tests 2020-02-16 11:31:35 -06:00
Giovanni Petrantoni ea0d167285 Fix max size write mplex test 2020-02-12 11:57:38 -05:00
Giovanni Petrantoni 20f65e0dc8 Use better async pattern in test (Notice this causes a issue to show up) 2020-02-12 11:57:38 -05:00