Commit Graph

50 Commits

Author SHA1 Message Date
Jacek Sieka 49a12e619d
channel close race and deadlock fixes (#368)
* channel close race and deadlock fixes

* remove send lock, write chunks in one go
* push some of half-closed implementation to BufferStream
* fix some hangs where LPChannel readers and writers would not always
wake up
* simplify lazy channels
* fix close happening more than once in some orderings
* reenable connection tracking tests
* close channels first on mplex close such that consumers can read bytes

A notable difference is that BufferedStream is no longer considered EOF
until someone has actually read the EOF marker.

* docs, simplification
2020-09-21 19:48:19 +02:00
Jacek Sieka 0db45462cd
mplex fixes (#362)
* remove almost-empty types module
* lock when writing message (that's the only place the lock matters, and
only when the message is > max msg size)
* logging updates (log in consistent order, makes reading logs easier)
* raise EOF from readExactly only if no bytes have been read (to signal
that _no_ bytes were lost)
2020-09-14 10:19:54 +02:00
Jacek Sieka c1856fda53
simplify and unify logging (#353)
* use short format for logging peerid
* log peerid:oid for connections
2020-09-06 10:31:47 +02:00
Jacek Sieka 397f9edfd4
simplify mplex (#327)
* less async
* less copying of data
* less redundant cleanup
2020-08-15 07:58:30 +02:00
Giovanni Petrantoni 5c986cf657
Fix build, add some raises (#315)
* Fix build, add some raises

* wip

* wip more raises

* missing exc object in mplex

* proper lifetime for subscribePeer

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-08-05 19:30:57 -06:00
Dmitriy Ryajov c788a6a3c0
Cleanup resources (#246)
* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises
2020-06-29 09:15:31 -06:00
Dmitriy Ryajov 5b28e8c488
Cleanup lpstream, Connection and BufferStream (#228)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* don't use delegation in connection

* move connection out to own file

* don't breakout on reset

* make sure to call close on secured conn

* add lpstream tracing

* don't breackdown by conn id

* fix import

* remove unused lable

* reset  connection on exception

* add additional metrics for skipped messages

* check for nil in secure.close
2020-06-19 11:29:43 -06:00
Viktor Kirilov 1afec627c2
proper name for topics so that we can filter dynamically using chronicles (#210)
* proper name for topics so that we can filter dynamically using chronicles

* lowercase
2020-06-10 10:48:01 +02:00
Dmitriy Ryajov 5960d42c50
remove casts from (#203) 2020-06-02 20:21:11 -06:00
Dmitriy Ryajov 4df151a3a3 typos 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov d83ce4c932 exceptions and resource cleanup 2020-06-02 09:10:27 -06:00
Giovanni Petrantoni 7c9e5c2f7a "Could not write to connection" message split between trace and debug due to log size 2020-05-30 23:47:56 +09:00
Dmitriy Ryajov 1819502fb5
Cleanup - tests and logging (#178)
* make async for proper exception handling

* tryAndWarn msg messes up Exception msg

* misc: comment out tracker dumps

* cleanup mplex tests

* more informative errors

* give CI time to run

* revert change, bacause it causes races
2020-05-18 07:49:49 -06:00
Jacek Sieka 69abf5097d
handle a few exceptions (#170)
* handle a few exceptions

Some of these are maybe too aggressive, but in return, they'll log
their exception - more refactoring needed to sort this out - right now
we get crashes on unhandled exceptions of unknown origin

* during connection setup
* while closing channels
* while processing pubsubs

* catch exceptions that are raised and don't try to catch exceptions that are not raised

* propagate cancellederror

* one more

* more

* more

* make interop tests less fragile

* Raise expiration time in gossipsub fanout test for slow CI

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-05-14 21:56:56 -06:00
Jacek Sieka 3053f03814 fix varint issues
* fixes #111
2020-05-11 09:12:23 -06:00
Jacek Sieka fbf640794b
move channel size constant 2020-05-09 08:48:26 +02:00
Giovanni Petrantoni e39bf0a4cf
Mplex: Add the ability to send any size payload (#123)
* Mplex: Add the ability to send any size payload

* Ensure size of coder header
2020-04-03 17:26:46 +02:00
Dmitriy Ryajov 5285f0d091
Fix/misc (#116)
* only check for payload size

* only subscribe if connection succeeded

* fix failing test

* check that the strem is active before openning

* msg type should not be > than 0x7

* fix tests

* check max against enum val
2020-03-29 08:28:48 -06:00
Dmitriy Ryajov 4199508f17 readMplexVarint should return uint64 2020-03-24 15:41:40 -06:00
Dmitriy Ryajov 2de98751ae fix: use exact types for mplex id 2020-03-24 15:41:40 -06:00
Dmitriy Ryajov 761b221b75 don't trace the payload 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov 5c234f704d added 1 byte jitter test 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov 406b79887d add jitter tolerance to mplex 2020-03-12 08:01:43 -06:00
Dmitriy Ryajov 82af623641 use correct shift 2020-03-11 09:58:24 -06:00
Dmitriy Ryajov 07695b2260 use `readExactly` not `read` to read all bytes 2020-03-11 09:58:24 -06:00
Dmitriy Ryajov 0349165830 typo 2020-02-16 11:31:35 -06:00
Dmitriy Ryajov 53e163abf2 check for size bounds when reading varints 2020-02-16 11:31:35 -06:00
Dmitriy Ryajov 94fc4e6fd2 don't use sleeps for synchronization 2020-02-16 11:31:35 -06:00
Zahary Karadjov 7bd305471c
Make sure the library can compile with json logging in trace mode 2020-02-04 15:17:39 +01:00
Dmitriy Ryajov 293a219dbe
Cleanup (#55)
* fix: don't allow replacing pubkey

* fix: several small improvements

* removing pubkey setter

* improove error handling

* remove the use of Option[T] if not needed

* don't use optional

* fix-ci: temporarily pin p2pd to a working tag

* fix example to comply with latest changes

* bumping p2pd again to a higher version
2019-12-10 14:50:35 -06:00
Dmitriy Ryajov e623e70e7b
PubSub (Gossip & Flood) Implementation (#36)
This adds gossipsub and floodsub, as well as basic interop testing with the go libp2p daemon. 

* add close event

* wip: gossipsub

* splitting rpc message

* making message handling more consistent

* initial gossipsub implementation

* feat: nim 1.0 cleanup

* wip: gossipsub protobuf

* adding encoding/decoding of gossipsub messages

* add disconnect handler

* add proper gossipsub msg handling

* misc: cleanup for nim 1.0

* splitting floodsub and gossipsub tests

* feat: add mesh rebalansing

* test pubsub

* add mesh rebalansing tests

* testing mesh maintenance

* finishing mcache implementatin

* wip: commenting out broken tests

* wip: don't run heartbeat for now

* switchout debug for trace logging

* testing gossip peer selection algorithm

* test stream piping

* more work around message amplification

* get the peerid from message

* use timed cache as backing store

* allow setting timeout in constructor

* several changes to improve performance

* more through testing of msg amplification

* prevent gc issues

* allow piping to self and prevent deadlocks

* improove floodsub

* allow running hook on cache eviction

* prevent race conditions

* prevent race conditions and improove tests

* use hashes as cache keys

* removing useless file

* don't create a new seq

* re-enable pubsub tests

* fix imports

* reduce number of runs to speed up tests

* break out control message processing

* normalize sleeps between steps

* implement proper transport filtering

* initial interop testing

* clean up floodsub publish logic

* allow dialing without a protocol

* adding multiple reads/writes

* use protobuf varint in mplex

* don't loose conn's peerInfo

* initial interop pubsub tests

* don't duplicate connections/peers

* bring back interop tests

* wip: interop

* re-enable interop and daemon tests

* add multiple read write tests from handlers

* don't cleanup channel prematurely

* use correct channel to send/receive msgs

* adjust tests with latest changes

* include interop tests

* remove temp logging output

* fix ci

* use correct public key serialization

* additional tests for pubsub interop
2019-12-05 20:16:18 -06:00
Dmitriy Ryajov 903e79ede1
Feat/conn cleanup (#41)
Backporting proper connection cleanup from #36 to align with latest chronos changes.

* add close event

* use proper varint encoding

* add proper channel cleanup in mplex

* add connection cleanup in secio

* tidy up

* add dollar operator

* fix tests

* don't close connections prematurely

* handle closing streams properly

* misc

* implement address filtering logic

* adding pipe tests

* don't use gcsafe if not needed

* misc

* proper connection cleanup and stream muxing

* re-enable pubsub tests
2019-12-03 22:44:54 -06:00
Dmitriy Ryajov f8872dd51c make log topics consistent 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov bc46a76029 connection should raise on invalid varint 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov f5508be544 yield to chronos in mplex loop 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 5ae37cac98 no more buffer resets on len prefixed reads 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 99869aa8b8 don't reset buffer on exception 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 9862064234 changed copyright year 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 054085620c logging: switch debug for trace in most cases 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov ef89e98479 renaming channel 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 8920cd7d60 misc: pubsub/floodsub and logging 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 74d2aea63d adding chronicles topics 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 435c69633f adding chronicles logging 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 701e048ee6 add debug logging 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 65ce1a93fc prevent interleaved channel data 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov e53c87e197 reworked to make msg reading sequential 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov e5b782f094 mplex testing/interop 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 8338a16aab testing mplex 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 9b485b3082 feat: half closed channels 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 6058a3fc69 split mplex 2019-10-11 08:15:24 +09:00