Commit Graph

111 Commits

Author SHA1 Message Date
Dmitriy Ryajov 9132f16927
gossipsub fixes (#186) 2020-05-21 14:24:20 -06:00
Dmitriy Ryajov ba53c08b3c
Track incoming connections (#181)
* call write until all is written out

* wip: rework with proper half-closed

* add eof and closed handling

* wip

* close connection on chronos close

* don't use read

* make noise work again

* don't reraise just yet

* fixes after backporting

* remove on transport close cleanup

* revert back allread

* rust interop fixes

* read from stream

* inc count before closing

* rebasing master

* store incomming connections

* fix merge

* remove unneeded changes

* use internal close flag to indicate disposal
2020-05-21 11:33:48 -06:00
Dmitriy Ryajov 7900fd9f61
Half closed (#174)
* call write until all is written out

* add comments to lpchannel fields

* add an eof flag to signal which end closed

* wip: rework with proper half-closed

* add eof and closed handling

* propagate closes to piped

* call parent close

* moving bufferstream trackers out

* move writeLock to bufferstream

* move writeLock out

* remove unused call

* wip

* rebasing master

* fix mplex tests

* wip

* fix bufferstream after backport

* wip

* rename to differentiate from chronos tracker

* close connection on chronos close

* make reset request asyncCheck

* fix channel cleanup

* misc

* don't use read

* fix backports

* make noise work again

* proper exception handling

* don't reraise just yet

* add convenience templates

* dont double wrap

* use async pragma

* fixes after backporting

* muxer owns connection

* remove on transport close cleanup

* revert back allread

* adding some todos

* read from stream

* inc count before closing

* rebasing master

* rebase master

* use correct exception type

* use try/finally insted of defer

* fix compile in trace mode

* reset channels on mplex close
2020-05-19 18:14:15 -06:00
Giovanni Petrantoni c219100e64
Use results and no exceptions in chacha and curve25519 (#182) 2020-05-19 10:22:49 +02:00
Dmitriy Ryajov f8029e7359
use sha256 digest as cache keys (#135)
* use sha256 digest as cache keys

* rebasing master
2020-05-18 14:49:49 -06:00
Giovanni Petrantoni 7dcb807f64
Crypto utilities resultification (#150) 2020-05-18 07:25:55 +02:00
Jacek Sieka 69abf5097d
handle a few exceptions (#170)
* handle a few exceptions

Some of these are maybe too aggressive, but in return, they'll log
their exception - more refactoring needed to sort this out - right now
we get crashes on unhandled exceptions of unknown origin

* during connection setup
* while closing channels
* while processing pubsubs

* catch exceptions that are raised and don't try to catch exceptions that are not raised

* propagate cancellederror

* one more

* more

* more

* make interop tests less fragile

* Raise expiration time in gossipsub fanout test for slow CI

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-05-14 21:56:56 -06:00
Jacek Sieka 3053f03814 fix varint issues
* fixes #111
2020-05-11 09:12:23 -06:00
Jacek Sieka 87e1f3c61f
fix trace logging 2020-05-09 13:56:07 +02:00
Dmitriy Ryajov 6196d56fc2
check for nil observers 2020-05-08 15:44:18 -06:00
Giovanni Petrantoni c889224012 Add PubSub observer+ hooks (they can modify as well) 2020-05-08 13:31:52 -06:00
Jacek Sieka 1efada474c
remove readLoop in secure protocols (#162)
* remove readLoop in secure protocols, fix security issues

* fix Defect on remote sending 0-byte noise/secio message
* remove msglen from `write` (unused)
* simplify SecureConn data flow
* document some control-flow issues

* unify exception behaviour across noise and secio

* secio would not raise on mac/decryption errors

* fix compile error
2020-05-07 14:37:46 -06:00
Jacek Sieka 330da51819
removals (#159)
* remove unused stream methods
* reimplement some of them with proc's
* remove broken tests
* Error->Defect for defect
* warning fixes
2020-05-06 18:31:47 +02:00
Dmitriy Ryajov 6da4d2af48
Pubsub signatures flags (#161)
* add verify signature flag

* add sign flag to enable/disable msg signing

* moving internal tests out to their own file

* cleanup nimble file

* remove unneeded tests

* move pubsub tests out

* fix tests
2020-05-06 11:26:08 +02:00
Giovanni Petrantoni 8a22c073c7 Fix secure/noise securing explicitly, added noise to pubsub tests 2020-04-24 14:33:08 -06:00
Giovanni Petrantoni 6ae92eb21a Hotfix noise read usage, replace with readExactly 2020-04-22 23:36:59 +09:00
Giovanni Petrantoni 4c6a123d31
Add chronos trackers and used them to sanitize resource disposal (#131)
* Add chronos trackers and used them to sanitize resource disposal

* Chronos trackers for transport tests wip

* No more chronos leaks in testtransport

* Make tcp transport and test more robust when closing

* Test async leaking tracking wip

* Fix a regression in wire connect

* Add chronos trackers to more tests and sanitize resource closure

* Wip fixing floodsub tests

* Floodsub wip

* Made floodsub basically deterministic, hit a nim bug with captures tho

* Wrap up floodsub tests refactor

* Wrapping up

* Add allFuturesThrowing utility

* Fix missing allFuturesThrowing in noise tests!

* Make tests green

* attempt fixing gossipsub failing cases

* Make sure to check also fanout in waitSub

* More verbose traces

* Gossipsub test improvments

* Refactor TcpTransport remove asyncCheck

* Add Connection trackers

* Add stricter connection tracking, wip mplex fix

* More asynccheck removal, in order to avoid connection leaks

* bump chronicles requirement

* Enable tracker dump to check CI output

* Wait for more futures in testmplex

* Remove tracker dump messages

* add tryAndWarn utility, fix mplex issue with go interop

* All allFuturesThrowing to directchat too

* make sure to cleanup on transport close
2020-04-21 10:24:42 +09:00
Ștefan Talpalaru eaa73ae6e8
add stream metrics (#136)
* add stream metrics

- just BufferStream and Connection are tracked, for now
- flag checking is enforced more strictly in close(), since it became
  clear that instances are closed multiple times

* add "metrics" dependency

and sort the list
2020-04-14 15:27:07 +02:00
Ștefan Talpalaru 7723403b1f
debug prints (#132)
* debug prints

* CI: enable stack traces

* Azure: better NimBinaries cache key

* CI changes

- Azure: remove Linux target
- Travis: add ARM64 target

* uglify the code in order to save 12 bytes per LPStream object
2020-04-14 15:21:16 +02:00
Giovanni Petrantoni 303ec297da
Start removing allFutures (#125)
* Start removing allFutures

* More allfutures removal

* Complete allFutures removal except legacy and tests

* Introduce table values copies to prevent error

* Switch to allFinished

* Resolve TODOs in flood/gossip

* muxer handler, log and re-raise

* Add a common and flexible way to check multiple futures
2020-04-11 13:08:25 +09:00
Dmitriy Ryajov 6cbcc7859e reduse usssage of asyncCheck 2020-04-07 12:16:59 -06:00
Dmitriy Ryajov 0c8dde15e7 remove unused field 2020-04-07 12:16:59 -06:00
Dmitriy Ryajov 87b35de83e close connections 2020-04-07 12:16:59 -06:00
Dmitriy Ryajov bae4713a85 small cleanup 2020-04-07 12:16:59 -06:00
Dmitriy Ryajov e4303110a6 properly close connections 2020-04-07 12:16:59 -06:00
Giovanni Petrantoni 35a48fa560 Re-enable gossipsub internal tests when running CI
minor bonus: add a link in the comments about bearssl issue with
callbacks
2020-04-07 22:07:00 +09:00
Giovanni Petrantoni 3514733060 Fix table assertion, edited while iterating
(the fix is not so nice.. adds plenty of allocations, but for now should
be ok)
2020-04-05 01:19:10 +09:00
Ștefan Talpalaru 7f8090b166
SecioConn.readMessage(): fix the exception types 2020-04-02 19:07:14 +02:00
Dmitriy Ryajov 3effb95f10 close underlying bufferstream in lpchannel 2020-03-28 09:29:43 -06:00
Giovanni Petrantoni 56392913f9
Allow to specify a read size in readLp (#117)
* Allow to specify a read size in readLp

Also fix secio fragments + test

* Revert readLp changes, use just `read` in test
2020-03-26 15:06:47 +09:00
Giovanni Petrantoni 33ceefb209 Remove nim random usage from noise 2020-03-26 15:04:46 +09:00
Giovanni Petrantoni b132eae939 Add secio oversized payload fragmentation
Cannot be used because Connection Read size hard limit tho
2020-03-25 16:53:35 +09:00
Giovanni Petrantoni 1550bea1cc Support arbitrary stream write sizes in noise 2020-03-25 16:10:11 +09:00
cheatfate 1f5d994700
Fix compilation errors introduced by latest chronos. 2020-03-24 09:48:05 +02:00
Giovanni Petrantoni 3d10513b3e Properly await the handshake end in secure handler 2020-03-24 15:34:02 +09:00
Giovanni Petrantoni 0a3e4a764b
Less verbose traces (#112)
* Make traces less verbose with shortHexDump utility

* Rename shortHexDump into shortLog

* Improve shortLog, add shortLog for crypto keys

* Add proper shortLog implementations in messages
2020-03-23 15:03:36 +09:00
Giovanni Petrantoni b1a34f478e Avoid 2 async consecutive writes in noise HS 2020-03-18 20:40:51 +09:00
Giovanni Petrantoni c02fca25f8
Noise (#90)
* Start ChaCha20Poly1305 integration (BearSSL)

* Add Curve25519 (BearSSL) required operations for noise

* Fix curve mulgen iterate/derive

* Fix misleading header

* Add chachapoly proper test

* Curve25519 integration tests (failing, something is wrong)

* Add few converters, finish c25519 integration tests

* Remove implicit converters

* removed internal globals

* Start noise implementation

* Fix public() using proper bear mulgen

* Noise protocol WIP

* Noise progress

* Add a quick nim version of HKDF

* Converted hkdf to iterator, useful for noise

* Noise protocol implementation progress

* Noise progress

* XX handshake almost there

* noise progress

* Noise, testing handshake with test vectors

* Noise handshake progress, still wrong somewhere!

* Noise handshake success!

* Full verified noise XX handshake completed

* Fix and rewrite test to be similar to switch one

* Start with connection upgrade

* Switch chachapoly to CT implementations

* Improve HKDF implementation

* Use a type insted of tuple for HandshakeResult

* Remove unnecessary Let

* More cosmetic fixes

* Properly check randomBytes result

* Fix chachapoly signature

* Noise full circle (altho dispatcher is nil cursed)

* Allow nil aads in chachapoly routines

* Noise implementation up to running full test

* Use bearssl HKDF as well

* Directly use bearssl rng for curve25519 keys

* Add a (disabled/no CI) noise interop test server

* WIP on fixing interop issues

* More fixes in noise implementation for interop

* bump chronos requirement (nimble)

* Add a chachapoly test for very small size payloads

* Noise, more tracing

* Add 2 properly working noise tests

* Fix payload packing, following the spec properly (and not go version but
rather rust)

* Sanity, replace discard with asyncCheck

* Small fixes and optimization

* Use stew endian2 rather then system endian module

* Update nimble deps (chronos)

* Minor cosmetic/code sanity fixes

* Noise, handle Nonce max

* Noise tests, make sure to close secured conns

* More polish, improve code readability too

* More polish and testing again which test fails

* Further polishing

* Restore noise tests

* Remove useless Future[void]

* Remove useless CipherState initializer

* add a proper read wait future in second noise test

* Remove noise generic secure implementation for now

* Few fixes to run eth2 sim

* Add more debug info in noise traces

* Merge size + payload write in sendEncryptedMessage

* Revert secure interface, add outgoing property directly in newNoise

* remove sendEncrypted and receiveEncrypted

* Use openarray in chachapoly and curve25519 helpers
2020-03-17 13:30:01 +01:00
Dmitriy Ryajov df29ac760e remove missing include 2020-03-09 13:26:38 -06:00
Dmitriy Ryajov fbe54e5c5a move SecureConn to secure.nim 2020-03-09 13:26:38 -06:00
Dmitriy Ryajov 19095a0f85 move `secure` to base as well 2020-03-09 13:26:38 -06:00
Eugene Kabanov d1c6591b8a Fix and refactoring of some procedures which are able to return nil as result (#97)
* Fix do not return nil as result.

* Fix mplex test to properly raise.
2020-03-09 13:26:38 -06:00
Dmitriy Ryajov c3e791db0d remove unused imports 2020-03-09 13:26:38 -06:00
Dmitriy Ryajov f1a297f0f3 add initiator flag to handshake 2020-03-09 13:26:38 -06:00
Dmitriy Ryajov d1d133319e change handleConn to be a proc 2020-03-09 13:26:38 -06:00
Dmitriy Ryajov 9aa9e97602 feat: move connection handling logic to base class 2020-03-09 13:26:38 -06:00
Dmitriy Ryajov 871d8d7478 remove white space 2020-03-09 13:26:38 -06:00
Eugene Kabanov 5701d937c8
Signed variable integers fixes. (#96)
* Fix signed varints.
Add tests for signed varints.
Remove some casts to allow usage at compile time.

* Fix vsizeof() on 32bit platforms.

* Add `hint` and `zint` types for proper signed integer encoding.

* Fix varint related bugs.

* Update requirements.

* Fix interop tests because of fixed readLine.

* Add putVarint, getVarint and tests.
2020-03-06 20:19:43 +01:00
Eugene Kabanov 381630f185
Fix and refactoring of some procedures which are able to return nil as result (#97)
* Fix do not return nil as result.

* Fix mplex test to properly raise.
2020-03-04 21:45:14 +02:00
Dmitriy Ryajov fbcef69891 implicitelly dial pubsub if enabled 2020-02-21 09:21:06 -06:00