63 Commits

Author SHA1 Message Date
Giovanni Petrantoni
8078fec0f0 Merge branch 'master' into gossip-one-one 2020-07-15 08:47:43 +09:00
Jacek Sieka
87e58c1c8d
metrics: one more pubsub peers fix 2020-07-13 16:16:46 +02:00
Jacek Sieka
c7895ccc52
metrics: fix pubsub_peers add metric 2020-07-13 16:15:27 +02:00
Giovanni Petrantoni
fcda0f6ce1
PubSubPeer tables refactor (#263)
* refactor peer tables

* tests fixing

* override PubSubPeer equality

* fix pubsubpeer comparison
2020-07-13 15:32:38 +02:00
Dmitriy Ryajov
4c815d75e7
More gossip cleanup (#257)
* more cleanup

* correct pubsub peer count

* close the stream first

* handle cancelation

* fix tests

* fix fanout ttl

* merging master

* remove `withLock` as it conflicts with stdlib

* fix trace build

Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-07-09 14:21:47 -06:00
Giovanni Petrantoni
f9e0a1f069 CI fix handleDisconnect (pubsub) 2020-07-09 13:56:59 +09:00
Giovanni Petrantoni
9b8b159abb Remove other spurious getStacktrace in pubsub traces 2020-07-09 13:19:34 +09:00
Giovanni Petrantoni
4698f41a91 Remove stacktrace logging from pubsub connect 2020-07-09 12:23:03 +09:00
Giovanni Petrantoni
f2e7035a17 fix gossip 11 upstream 2020-07-08 19:22:02 +09:00
Giovanni Petrantoni
38422ff4cb Merge branch 'master' into gossip-one-one 2020-07-08 11:39:33 +09:00
Dmitriy Ryajov
a52763cc6d
fix publishing (#250)
* use var semantics to optimize table access

* wip... lvalues don't work properly sadly...

* big publish refactor, replenish and balance

* fix internal tests

* use g.peers for fanout (todo: don't include flood peers)

* exclude non gossip from fanout

* internal test fixes

* fix flood tests

* fix test's trypublish

* test interop fixes

* make sure to not remove peers from gossip table

* restore old replenishFanout

* cleanups

* Cleanup resources (#246)

* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises

* restore trace vs debug in gossip

* improve fanout replenish behavior further

* cleanup stale peers more eaguerly

* synchronize connection cleanup and small refactor

* close client first and call parent second

* disconnect failed peers on publish

* check for publish result

* fix tests

* fix tests

* always call close

Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-07-07 18:33:05 -06:00
Giovanni Petrantoni
2f39cf6f42 Merge branch 'publish-backporting' into gossip-one-one 2020-07-04 19:37:31 +09:00
Giovanni Petrantoni
ad7db1ca26 add more heartbeat locking to prevent races 2020-07-04 14:04:44 +09:00
Giovanni Petrantoni
466f3c9171 add back topics in peers, re-enable flood publish 2020-07-04 12:15:35 +09:00
Giovanni Petrantoni
be0375f840 fix up merge 2020-07-04 11:51:09 +09:00
Giovanni Petrantoni
afb5cb88f9 Merge remote-tracking branch 'origin/publish-backporting' into gossip-one-one 2020-07-04 11:42:16 +09:00
Giovanni Petrantoni
0d0309a601 better diagnostics 2020-07-03 09:28:20 +09:00
Giovanni Petrantoni
5e4934bcce Merge branch 'master' into publish-backporting 2020-07-02 17:44:23 +09:00
Giovanni Petrantoni
4f5c97e27c remove topics from pubsubpeer (was unused) 2020-07-01 17:44:26 +09:00
Giovanni Petrantoni
ec00c7fc50
Peer resultification and defect only (#245)
* Peer resultification and defect only

* Fixing some tests

* test fixes

* Rename peer into peerid

* better result error message in identify

* further merge fixes
2020-07-01 08:25:09 +02:00
Dmitriy Ryajov
c788a6a3c0
Cleanup resources (#246)
* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises
2020-06-29 09:15:31 -06:00
Giovanni Petrantoni
3a657135dc Merge branch 'master' into publish-backporting 2020-06-29 22:02:58 +09:00
Jacek Sieka
aa6756dfe0
allow message id provider to be specified (#243)
* don't send public key in message when not signing (information leak)
* don't run rebalance if there are peers in gossip (see #242)
* don't crash randomly on bad peer id from remote
2020-06-28 09:56:38 -06:00
Giovanni Petrantoni
8e2c9301a0 use g.peers for fanout (todo: don't include flood peers) 2020-06-28 10:53:43 +09:00
Giovanni Petrantoni
dc3c568dc0 wip... lvalues don't work properly sadly... 2020-06-27 14:59:48 +09:00
Giovanni Petrantoni
98e4bb6769 add a way to pass parameters to gossip 2020-06-25 21:14:22 +09:00
Giovanni Petrantoni
3568b073fd allow multiple codecs per protocol (without breaking things) 2020-06-23 11:48:11 +09:00
Dmitriy Ryajov
5b28e8c488
Cleanup lpstream, Connection and BufferStream (#228)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* don't use delegation in connection

* move connection out to own file

* don't breakout on reset

* make sure to call close on secured conn

* add lpstream tracing

* don't breackdown by conn id

* fix import

* remove unused lable

* reset  connection on exception

* add additional metrics for skipped messages

* check for nil in secure.close
2020-06-19 11:29:43 -06:00
Dmitriy Ryajov
fe828d87d8
count published messages (#224) 2020-06-16 22:14:02 -06:00
Dmitriy Ryajov
ac04ca6e31
make sure keys exist and more metrics (#215) 2020-06-11 20:20:58 -06:00
Dmitriy Ryajov
6b196ad7b4
remove pubsub peer on disconnect (#212)
* remove pubsub peer on disconnect

* make sure lock is aquired

* add $

* count upgrades/dials/disconnects
2020-06-11 08:45:59 -06:00
Viktor Kirilov
1afec627c2
proper name for topics so that we can filter dynamically using chronicles (#210)
* proper name for topics so that we can filter dynamically using chronicles

* lowercase
2020-06-10 10:48:01 +02:00
Giovanni Petrantoni
82b4ed8f44 use declareCounter rather then gauge for certain metrics 2020-06-07 16:41:23 +09:00
Giovanni Petrantoni
a6a2a81711
Start adding some metrics to pubsub (#192)
* Start adding some metrics to pubsub

In order to visualize it's functionality
Still WIP

* more metrics

* add per topic metrics

* finishup with requested metrics

* add a metrisServer define to start local server

* PR fixes and cleanup
2020-06-07 09:15:21 +02:00
Dmitriy Ryajov
293b7da295 typo 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov
93e5805c01 better exception handling 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov
d4bdb42046 gossipsub fixes 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov
7b6e1c0688
Gossipsub interop (#189)
* interop fixes

* add custom messageid provider and fix seqno

* use ECDSA for speed

* adding messageid tests

* breakout from publish loop

* addressing review comments

* remove unneded var

* dont stop broadcasting on failed peers
2020-05-27 12:33:49 -06:00
Dmitriy Ryajov
9132f16927
gossipsub fixes (#186) 2020-05-21 14:24:20 -06:00
Dmitriy Ryajov
ba53c08b3c
Track incoming connections (#181)
* call write until all is written out

* wip: rework with proper half-closed

* add eof and closed handling

* wip

* close connection on chronos close

* don't use read

* make noise work again

* don't reraise just yet

* fixes after backporting

* remove on transport close cleanup

* revert back allread

* rust interop fixes

* read from stream

* inc count before closing

* rebasing master

* store incomming connections

* fix merge

* remove unneeded changes

* use internal close flag to indicate disposal
2020-05-21 11:33:48 -06:00
Jacek Sieka
69abf5097d
handle a few exceptions (#170)
* handle a few exceptions

Some of these are maybe too aggressive, but in return, they'll log
their exception - more refactoring needed to sort this out - right now
we get crashes on unhandled exceptions of unknown origin

* during connection setup
* while closing channels
* while processing pubsubs

* catch exceptions that are raised and don't try to catch exceptions that are not raised

* propagate cancellederror

* one more

* more

* more

* make interop tests less fragile

* Raise expiration time in gossipsub fanout test for slow CI

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-05-14 21:56:56 -06:00
Giovanni Petrantoni
c889224012 Add PubSub observer+ hooks (they can modify as well) 2020-05-08 13:31:52 -06:00
Jacek Sieka
330da51819
removals (#159)
* remove unused stream methods
* reimplement some of them with proc's
* remove broken tests
* Error->Defect for defect
* warning fixes
2020-05-06 18:31:47 +02:00
Dmitriy Ryajov
6da4d2af48
Pubsub signatures flags (#161)
* add verify signature flag

* add sign flag to enable/disable msg signing

* moving internal tests out to their own file

* cleanup nimble file

* remove unneeded tests

* move pubsub tests out

* fix tests
2020-05-06 11:26:08 +02:00
Giovanni Petrantoni
303ec297da
Start removing allFutures (#125)
* Start removing allFutures

* More allfutures removal

* Complete allFutures removal except legacy and tests

* Introduce table values copies to prevent error

* Switch to allFinished

* Resolve TODOs in flood/gossip

* muxer handler, log and re-raise

* Add a common and flexible way to check multiple futures
2020-04-11 13:08:25 +09:00
Dmitriy Ryajov
6cbcc7859e reduse usssage of asyncCheck 2020-04-07 12:16:59 -06:00
cheatfate
1f5d994700
Fix compilation errors introduced by latest chronos. 2020-03-24 09:48:05 +02:00
Giovanni Petrantoni
0a3e4a764b
Less verbose traces (#112)
* Make traces less verbose with shortHexDump utility

* Rename shortHexDump into shortLog

* Improve shortLog, add shortLog for crypto keys

* Add proper shortLog implementations in messages
2020-03-23 15:03:36 +09:00
Dmitriy Ryajov
fbcef69891 implicitelly dial pubsub if enabled 2020-02-21 09:21:06 -06:00
Zahary Karadjov
7bd305471c
Make sure the library can compile with json logging in trace mode 2020-02-04 15:17:39 +01:00