Commit Graph

592 Commits

Author SHA1 Message Date
Giovanni Petrantoni ec00c7fc50
Peer resultification and defect only (#245)
* Peer resultification and defect only

* Fixing some tests

* test fixes

* Rename peer into peerid

* better result error message in identify

* further merge fixes
2020-07-01 08:25:09 +02:00
Dmitriy Ryajov c788a6a3c0
Cleanup resources (#246)
* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises
2020-06-29 09:15:31 -06:00
Jacek Sieka aa6756dfe0
allow message id provider to be specified (#243)
* don't send public key in message when not signing (information leak)
* don't run rebalance if there are peers in gossip (see #242)
* don't crash randomly on bad peer id from remote
2020-06-28 09:56:38 -06:00
Dmitriy Ryajov 902880ef1f
consolidate reading in lpstream (#241)
* consolidate reading in lpstream

* remove debug echo

* throw if not enough bytes where read

* tune log level

* set eof flag

* test readExactly to fail on not enough bytes
2020-06-27 11:33:34 -06:00
Dmitriy Ryajov 7a95f1844b
Concurrent dials (#238)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* add secureconn constructor

* close in the correct order

* concurent dial lock and track in/out conns better

* make tests pass

* add todo comment

* disconect peers that open too many connections

* wip

* do connection and muxer tracking in one place

* prevent nil pointer in observers

* drop connections when peers is over max

* prevent channel leaks

* don't use closure to handle channel
2020-06-24 09:08:44 -06:00
Giovanni Petrantoni ee6e545878
multistream select make sure to not report NA (#235)
* multistream select make sure to not report NA but rather empty string if all fails

Also re-enable tests

* avoid using bad constructs, make multistream.select flow crystal clear
2020-06-22 15:38:48 -06:00
Jacek Sieka 6331b04cb4
secp: requiresInit updates (#237)
* secp: requiresInit updates

* fix
2020-06-22 19:03:15 +02:00
Jacek Sieka b99fd88deb
logging fixes 2020-06-21 11:14:19 +02:00
Giovanni Petrantoni 7852c6dd0f
Noise and eth2/nbc fixes (#226)
* Remove noise padding payload (spec removed it)

* add log scope in secure

* avoid defect array out of range in switch secure when "na"

* improve identify traces

* wip noise fixes

* noise protobuf adjustments (trying)

* add more debugging messages/traces, improve their actual contents

* re-enable ID check in noise

* bump go daemon tag version

* bump go daemon tag version

* enable noise in daemonapi

* interop testing, (both secio and noise will be tested)

* azure cache bump (p2pd)

* CI changes

- Travis: use Go 1.14
- azure-pipelines.yml: big cleanup
- Azure: bump cache keys
- build 64-bit p2pd on 32-bit Windows
- install both Mingw-w64 architectures

* noise logging fixes

* alternate testing between noise and secio

* increase timeout to avoid VM errors in CI (multistream tests)

* refactor heartbeat management in gossipsub

* remove locking within heartbeat

* refactor heartbeat management in gossipsub

* remove locking within heartbeat

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2020-06-20 19:56:55 +09:00
Dmitriy Ryajov 7a1c1c2ea6
fixing some key not found exceptions (#231) 2020-06-19 15:19:07 -06:00
Dmitriy Ryajov 5b28e8c488
Cleanup lpstream, Connection and BufferStream (#228)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* don't use delegation in connection

* move connection out to own file

* don't breakout on reset

* make sure to call close on secured conn

* add lpstream tracing

* don't breackdown by conn id

* fix import

* remove unused lable

* reset  connection on exception

* add additional metrics for skipped messages

* check for nil in secure.close
2020-06-19 11:29:43 -06:00
Dmitriy Ryajov 719744f46a
Small fixes (#230)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* make sure sending doesn't fail

* add `contains`

* review comment from prev pr
2020-06-19 11:29:25 -06:00
Dmitriy Ryajov 5f75e5adc6
don't use `switch.dial` in `subscribeToPeer` (#227)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`
2020-06-17 20:12:32 -06:00
Dmitriy Ryajov fe828d87d8
count published messages (#224) 2020-06-16 22:14:02 -06:00
Giovanni Petrantoni 224c2613e3
Remove noise padding payload (spec removed it) (#222) 2020-06-16 17:34:02 -06:00
Jacek Sieka 523eed65ce
metrics: use eth2 name for peers metric (#223) 2020-06-16 14:43:45 -06:00
Giovanni Petrantoni 0d81acafa8 Fix multistream select trace proto logging 2020-06-16 12:02:47 +09:00
Dmitriy Ryajov 9d9f793b4f
add metrics for sent messages by topic and peer (#220) 2020-06-15 17:39:03 -06:00
Dmitriy Ryajov 3c4c10d871
con't crash on nil ptr 2020-06-15 13:37:25 -06:00
Dmitriy Ryajov 7cb6c81159
Don't modify iterables while iterating them (#219)
* don't modify iterables while iterating

* assert handlers to properly close connections
2020-06-15 12:30:09 -06:00
Dmitriy Ryajov 9cd47fe816
add a logScope to make tracing less ugly (#218)
* add a logScope to make tracing less ugly

* don't crash on nil pointer
2020-06-15 12:10:41 -06:00
Dmitriy Ryajov 85b56d0b3a
Observedaddr (#217)
* send correct observerAddr

* run tests with info log level

* set observedaddr for channels
2020-06-12 19:56:17 -06:00
Oskar Thorén f97e2deec7
Make methods in FloodSub, GossipSub public (#216)
Similar to https://github.com/status-im/nim-libp2p/pull/193 but doing it for all
methods to avoid this issue in PubSub in the future. Got hit by this behavior
again in rpcHandler and took me a while to figure out.

See https://github.com/oskarth/nim-private-method-skipping-case/ for minimal
repro
2020-06-12 17:54:12 -06:00
Dmitriy Ryajov ac04ca6e31
make sure keys exist and more metrics (#215) 2020-06-11 20:20:58 -06:00
Dmitriy Ryajov 55a294a5c9
better pubsub metrics (#214) 2020-06-11 12:09:34 -06:00
Dmitriy Ryajov 6b196ad7b4
remove pubsub peer on disconnect (#212)
* remove pubsub peer on disconnect

* make sure lock is aquired

* add $

* count upgrades/dials/disconnects
2020-06-11 08:45:59 -06:00
Jacek Sieka 92579435b6
secio then noise (#213)
* secio then noise

much fewer peers on witti with noise first

* comment
2020-06-11 08:38:47 +02:00
Viktor Kirilov 1afec627c2
proper name for topics so that we can filter dynamically using chronicles (#210)
* proper name for topics so that we can filter dynamically using chronicles

* lowercase
2020-06-10 10:48:01 +02:00
Jacek Sieka 8d9e231a74
grab agentversion/protoversion (#211) 2020-06-09 12:42:52 -06:00
Dmitriy Ryajov 35ff99829e
close streams (#208)
* close streams

* don't warn on missing proto
2020-06-08 17:42:27 -06:00
Dmitriy Ryajov ee281310c0
move trace log 2020-06-08 10:40:08 -06:00
Giovanni Petrantoni 82b4ed8f44 use declareCounter rather then gauge for certain metrics 2020-06-07 16:41:23 +09:00
Giovanni Petrantoni a6a2a81711
Start adding some metrics to pubsub (#192)
* Start adding some metrics to pubsub

In order to visualize it's functionality
Still WIP

* more metrics

* add per topic metrics

* finishup with requested metrics

* add a metrisServer define to start local server

* PR fixes and cleanup
2020-06-07 09:15:21 +02:00
Dmitriy Ryajov 130c64f33a
don't return nil in dial (#205)
* dont return nil in dial

* don't crash on pubsub send
2020-06-05 18:17:05 -06:00
Zahary Karadjov 2aebae56c0
Don't rely on the side-effects from doAssert 2020-06-05 19:12:10 +03:00
Zahary Karadjov 828a80ec8f
Make the MultiAddress.init function used in NBC non-failing 2020-06-05 17:51:22 +03:00
Dmitriy Ryajov 5960d42c50
remove casts from (#203) 2020-06-02 20:21:11 -06:00
Dmitriy Ryajov bb8bff2195
add sparse message propagation tests to gossipsub (#202)
* add sparce tests to gossipsub

* add send hooks

* remove `all`
2020-06-02 17:53:38 -06:00
Dmitriy Ryajov 285884c20c
Close peers (#201)
* wip

* exceptions and resource cleanup

* correct peerlifetime on disconnect

* emulate defered

* remove comment
2020-06-02 11:32:42 -06:00
Dmitriy Ryajov c7298f34f4 additional comments 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 71640da8f2 remove close from read/write methods 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 1b4876d26d emulate `defered` 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov abf659a01a more consistent dialing proto selecting logic 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 86e1c8169c decorate observers hooks with {.raises: [Defect].}
move hooks logic out into standalone procs

License: MIT
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 4df151a3a3 typos 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 27a7f8c948 move EOF flag after local close and comments 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 815282a5da remove all() 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 293b7da295 typo 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 6e0eb93d4f remove readloops 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov abc12d0fb5 move stram close to a better location 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov daef00fc7b don't crash schlesi-dev 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 640c3bdc45 better exception handling and resource cleanup 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 7ff76d76b6 better exceptions 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 95774b2b81 better cleanup 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 9c2f31262e wip: try handling child stream exceptions 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov e3f8f53620 initStream method and better exceptions handling 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov d3b79b002e better exceptions and don't fail writes 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 74f8b5b5f1 resource cleanup 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 93e5805c01 better exception handling 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov d83ce4c932 exceptions and resource cleanup 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov d4bdb42046 gossipsub fixes 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 9d3cc9647b fix merge 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 3d9c0bffba read from stream 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 048b1db1ad revert back allread 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 8e9716f5c3 remove on transport close cleanup 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov cf76edb0dd make noise work again 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 5158d96eaf close connection on chronos close 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 46daed9a38 wip 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 0f691cbafd add eof and closed handling 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 62da2a05c3 wip: rework with proper half-closed 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 98117a3068 call write until all is written out 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 802299e69a breakout from publish loop 2020-06-02 09:10:27 -06:00
Jacek Sieka 7e31210455
cover missing case in MultiAddress.init (#198)
* cover missing case in MultiAddress.init

* raise assert on marker in protocol

* unify inits for markers / non-markers

* fix string
2020-06-01 14:41:45 +02:00
Jacek Sieka 88dbeebf17
add side effect annotations (#197) 2020-06-01 09:25:16 +02:00
Giovanni Petrantoni 37b98ad45c
Secure managers are now sorted, giving priority to noise (#191)
* Secure managers are now sorted, giving priority to noise

* fix nimble test command

* Fix native tests

* fix directchat sample

* Could not write to connection - reduce verbosity

* fix interop testing

* Remove more tables

* test interop fixes

* directchat fix

* fix interop/remove some deprecation
2020-06-01 08:41:32 +02:00
Giovanni Petrantoni 6affcda937
Less exceptions more results (#188)
* Less exceptions more results

* Fix daemonapi and interop tests

* Add multibase

* wip multiaddress

* fix the build, consuming new result types

* fix standard setup

* Simplify match, rename into MaError, add more exaustive err text

* Fix the CI issues

* Fix directchat build

* daemon api fixes

* better err messages formatting

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-05-31 16:22:49 +02:00
Giovanni Petrantoni 7c9e5c2f7a "Could not write to connection" message split between trace and debug due to log size 2020-05-30 23:47:56 +09:00
Oskar Thorén b88bfc05f8
Make GossipSub initPubSub method public (#193)
This means we can use it from other protocols that inherit GossipSub. Otherwise,
a lot of internal state (heartbeat lock etc) doesn't get initialized properly.
2020-05-29 09:35:03 -06:00
Dmitriy Ryajov 7b6e1c0688
Gossipsub interop (#189)
* interop fixes

* add custom messageid provider and fix seqno

* use ECDSA for speed

* adding messageid tests

* breakout from publish loop

* addressing review comments

* remove unneded var

* dont stop broadcasting on failed peers
2020-05-27 12:33:49 -06:00
Giovanni Petrantoni 536555138c fix "future" typo 2020-05-26 16:08:09 +09:00
Giovanni Petrantoni 4447d97234 add back forgotten message in tryAndWarn 2020-05-26 15:21:59 +09:00
Dmitriy Ryajov 9132f16927
gossipsub fixes (#186) 2020-05-21 14:24:20 -06:00
Dmitriy Ryajov ba53c08b3c
Track incoming connections (#181)
* call write until all is written out

* wip: rework with proper half-closed

* add eof and closed handling

* wip

* close connection on chronos close

* don't use read

* make noise work again

* don't reraise just yet

* fixes after backporting

* remove on transport close cleanup

* revert back allread

* rust interop fixes

* read from stream

* inc count before closing

* rebasing master

* store incomming connections

* fix merge

* remove unneeded changes

* use internal close flag to indicate disposal
2020-05-21 11:33:48 -06:00
Dmitriy Ryajov 7900fd9f61
Half closed (#174)
* call write until all is written out

* add comments to lpchannel fields

* add an eof flag to signal which end closed

* wip: rework with proper half-closed

* add eof and closed handling

* propagate closes to piped

* call parent close

* moving bufferstream trackers out

* move writeLock to bufferstream

* move writeLock out

* remove unused call

* wip

* rebasing master

* fix mplex tests

* wip

* fix bufferstream after backport

* wip

* rename to differentiate from chronos tracker

* close connection on chronos close

* make reset request asyncCheck

* fix channel cleanup

* misc

* don't use read

* fix backports

* make noise work again

* proper exception handling

* don't reraise just yet

* add convenience templates

* dont double wrap

* use async pragma

* fixes after backporting

* muxer owns connection

* remove on transport close cleanup

* revert back allread

* adding some todos

* read from stream

* inc count before closing

* rebasing master

* rebase master

* use correct exception type

* use try/finally insted of defer

* fix compile in trace mode

* reset channels on mplex close
2020-05-19 18:14:15 -06:00
Dmitriy Ryajov 681991ae48
reduce buffer size in lpchannel (#184) 2020-05-19 16:15:36 -06:00
Giovanni Petrantoni 01339c991f
Don't use and expose directly secp types (#183)
* Don't use and expose directly secp types

* Reuse same secp type names
2020-05-19 14:48:55 +02:00
Giovanni Petrantoni c219100e64
Use results and no exceptions in chacha and curve25519 (#182) 2020-05-19 10:22:49 +02:00
Dmitriy Ryajov f8029e7359
use sha256 digest as cache keys (#135)
* use sha256 digest as cache keys

* rebasing master
2020-05-18 14:49:49 -06:00
Dmitriy Ryajov 9cf1fd0216
remove generic constructor and expose serverflags (#176)
* remove generic constructor and expose serverflags

* fix transport constructor

* fix merge issues
2020-05-18 13:04:05 -06:00
Dmitriy Ryajov 773b738c12
don't track Connection, track StreamTransport (#177)
* don't track Connection, track StreamTransport

* make tests more deterministic
2020-05-18 11:05:34 -06:00
Dmitriy Ryajov 5583168965
Track all connections (#175)
* adding to do for future refactor

* no mix conn and transport stopping or it will race

* don't use intermediary vars
2020-05-18 09:08:10 -06:00
Dmitriy Ryajov 1819502fb5
Cleanup - tests and logging (#178)
* make async for proper exception handling

* tryAndWarn msg messes up Exception msg

* misc: comment out tracker dumps

* cleanup mplex tests

* more informative errors

* give CI time to run

* revert change, bacause it causes races
2020-05-18 07:49:49 -06:00
Giovanni Petrantoni 7dcb807f64
Crypto utilities resultification (#150) 2020-05-18 07:25:55 +02:00
Dmitriy Ryajov 167f42ed45
Remove read (#171)
* use readExactly

* remove `read`

* remove read

* no more `read`
2020-05-14 22:02:05 -06:00
Jacek Sieka 69abf5097d
handle a few exceptions (#170)
* handle a few exceptions

Some of these are maybe too aggressive, but in return, they'll log
their exception - more refactoring needed to sort this out - right now
we get crashes on unhandled exceptions of unknown origin

* during connection setup
* while closing channels
* while processing pubsubs

* catch exceptions that are raised and don't try to catch exceptions that are not raised

* propagate cancellederror

* one more

* more

* more

* make interop tests less fragile

* Raise expiration time in gossipsub fanout test for slow CI

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-05-14 21:56:56 -06:00
Giovanni Petrantoni 005e088405
Properly track and close mplex handlers (#166)
* Properly track and close mplex handlers

* Avoid verbose warnings

* Fix tryAndWarn trace issue

* Handle LPEOF in lpchannel close
2020-05-12 14:45:32 +02:00
Jacek Sieka 618e01eba3
don't crash on double peerinfo close (#167) 2020-05-11 21:05:24 +02:00
Jacek Sieka 3053f03814 fix varint issues
* fixes #111
2020-05-11 09:12:23 -06:00
Jacek Sieka 87e1f3c61f
fix trace logging 2020-05-09 13:56:07 +02:00
Jacek Sieka fbf640794b
move channel size constant 2020-05-09 08:48:26 +02:00