Commit Graph

49 Commits

Author SHA1 Message Date
Giovanni Petrantoni be0375f840 fix up merge 2020-07-04 11:51:09 +09:00
Giovanni Petrantoni afb5cb88f9 Merge remote-tracking branch 'origin/publish-backporting' into gossip-one-one 2020-07-04 11:42:16 +09:00
Giovanni Petrantoni 0d0309a601 better diagnostics 2020-07-03 09:28:20 +09:00
Giovanni Petrantoni 5e4934bcce Merge branch 'master' into publish-backporting 2020-07-02 17:44:23 +09:00
Giovanni Petrantoni 4f5c97e27c remove topics from pubsubpeer (was unused) 2020-07-01 17:44:26 +09:00
Giovanni Petrantoni ec00c7fc50
Peer resultification and defect only (#245)
* Peer resultification and defect only

* Fixing some tests

* test fixes

* Rename peer into peerid

* better result error message in identify

* further merge fixes
2020-07-01 08:25:09 +02:00
Dmitriy Ryajov c788a6a3c0
Cleanup resources (#246)
* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises
2020-06-29 09:15:31 -06:00
Giovanni Petrantoni 3a657135dc Merge branch 'master' into publish-backporting 2020-06-29 22:02:58 +09:00
Jacek Sieka aa6756dfe0
allow message id provider to be specified (#243)
* don't send public key in message when not signing (information leak)
* don't run rebalance if there are peers in gossip (see #242)
* don't crash randomly on bad peer id from remote
2020-06-28 09:56:38 -06:00
Giovanni Petrantoni 8e2c9301a0 use g.peers for fanout (todo: don't include flood peers) 2020-06-28 10:53:43 +09:00
Giovanni Petrantoni dc3c568dc0 wip... lvalues don't work properly sadly... 2020-06-27 14:59:48 +09:00
Giovanni Petrantoni 98e4bb6769 add a way to pass parameters to gossip 2020-06-25 21:14:22 +09:00
Giovanni Petrantoni 3568b073fd allow multiple codecs per protocol (without breaking things) 2020-06-23 11:48:11 +09:00
Dmitriy Ryajov 5b28e8c488
Cleanup lpstream, Connection and BufferStream (#228)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* don't use delegation in connection

* move connection out to own file

* don't breakout on reset

* make sure to call close on secured conn

* add lpstream tracing

* don't breackdown by conn id

* fix import

* remove unused lable

* reset  connection on exception

* add additional metrics for skipped messages

* check for nil in secure.close
2020-06-19 11:29:43 -06:00
Dmitriy Ryajov fe828d87d8
count published messages (#224) 2020-06-16 22:14:02 -06:00
Dmitriy Ryajov ac04ca6e31
make sure keys exist and more metrics (#215) 2020-06-11 20:20:58 -06:00
Dmitriy Ryajov 6b196ad7b4
remove pubsub peer on disconnect (#212)
* remove pubsub peer on disconnect

* make sure lock is aquired

* add $

* count upgrades/dials/disconnects
2020-06-11 08:45:59 -06:00
Viktor Kirilov 1afec627c2
proper name for topics so that we can filter dynamically using chronicles (#210)
* proper name for topics so that we can filter dynamically using chronicles

* lowercase
2020-06-10 10:48:01 +02:00
Giovanni Petrantoni 82b4ed8f44 use declareCounter rather then gauge for certain metrics 2020-06-07 16:41:23 +09:00
Giovanni Petrantoni a6a2a81711
Start adding some metrics to pubsub (#192)
* Start adding some metrics to pubsub

In order to visualize it's functionality
Still WIP

* more metrics

* add per topic metrics

* finishup with requested metrics

* add a metrisServer define to start local server

* PR fixes and cleanup
2020-06-07 09:15:21 +02:00
Dmitriy Ryajov 293b7da295 typo 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 93e5805c01 better exception handling 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov d4bdb42046 gossipsub fixes 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 7b6e1c0688
Gossipsub interop (#189)
* interop fixes

* add custom messageid provider and fix seqno

* use ECDSA for speed

* adding messageid tests

* breakout from publish loop

* addressing review comments

* remove unneded var

* dont stop broadcasting on failed peers
2020-05-27 12:33:49 -06:00
Dmitriy Ryajov 9132f16927
gossipsub fixes (#186) 2020-05-21 14:24:20 -06:00
Dmitriy Ryajov ba53c08b3c
Track incoming connections (#181)
* call write until all is written out

* wip: rework with proper half-closed

* add eof and closed handling

* wip

* close connection on chronos close

* don't use read

* make noise work again

* don't reraise just yet

* fixes after backporting

* remove on transport close cleanup

* revert back allread

* rust interop fixes

* read from stream

* inc count before closing

* rebasing master

* store incomming connections

* fix merge

* remove unneeded changes

* use internal close flag to indicate disposal
2020-05-21 11:33:48 -06:00
Jacek Sieka 69abf5097d
handle a few exceptions (#170)
* handle a few exceptions

Some of these are maybe too aggressive, but in return, they'll log
their exception - more refactoring needed to sort this out - right now
we get crashes on unhandled exceptions of unknown origin

* during connection setup
* while closing channels
* while processing pubsubs

* catch exceptions that are raised and don't try to catch exceptions that are not raised

* propagate cancellederror

* one more

* more

* more

* make interop tests less fragile

* Raise expiration time in gossipsub fanout test for slow CI

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-05-14 21:56:56 -06:00
Giovanni Petrantoni c889224012 Add PubSub observer+ hooks (they can modify as well) 2020-05-08 13:31:52 -06:00
Jacek Sieka 330da51819
removals (#159)
* remove unused stream methods
* reimplement some of them with proc's
* remove broken tests
* Error->Defect for defect
* warning fixes
2020-05-06 18:31:47 +02:00
Dmitriy Ryajov 6da4d2af48
Pubsub signatures flags (#161)
* add verify signature flag

* add sign flag to enable/disable msg signing

* moving internal tests out to their own file

* cleanup nimble file

* remove unneeded tests

* move pubsub tests out

* fix tests
2020-05-06 11:26:08 +02:00
Giovanni Petrantoni 303ec297da
Start removing allFutures (#125)
* Start removing allFutures

* More allfutures removal

* Complete allFutures removal except legacy and tests

* Introduce table values copies to prevent error

* Switch to allFinished

* Resolve TODOs in flood/gossip

* muxer handler, log and re-raise

* Add a common and flexible way to check multiple futures
2020-04-11 13:08:25 +09:00
Dmitriy Ryajov 6cbcc7859e reduse usssage of asyncCheck 2020-04-07 12:16:59 -06:00
cheatfate 1f5d994700
Fix compilation errors introduced by latest chronos. 2020-03-24 09:48:05 +02:00
Giovanni Petrantoni 0a3e4a764b
Less verbose traces (#112)
* Make traces less verbose with shortHexDump utility

* Rename shortHexDump into shortLog

* Improve shortLog, add shortLog for crypto keys

* Add proper shortLog implementations in messages
2020-03-23 15:03:36 +09:00
Dmitriy Ryajov fbcef69891 implicitelly dial pubsub if enabled 2020-02-21 09:21:06 -06:00
Zahary Karadjov 7bd305471c
Make sure the library can compile with json logging in trace mode 2020-02-04 15:17:39 +01:00
Dmitriy Ryajov 667691f784 send messages in batches 2020-01-09 12:55:21 -06:00
Dmitriy Ryajov 0fb1f1c5b8 strenghten pubsub interop testing 2019-12-24 10:35:35 -06:00
Dmitriy Ryajov 68cc57669e
Feat/pubsub validators (#58)
* feat: adding validator hooks to pubsub

* expose add/remove validators on switch

* do less unnecessary copyng
2019-12-16 23:24:03 -06:00
Dmitriy Ryajov 293a219dbe
Cleanup (#55)
* fix: don't allow replacing pubkey

* fix: several small improvements

* removing pubkey setter

* improove error handling

* remove the use of Option[T] if not needed

* don't use optional

* fix-ci: temporarily pin p2pd to a working tag

* fix example to comply with latest changes

* bumping p2pd again to a higher version
2019-12-10 14:50:35 -06:00
Dmitriy Ryajov 5f6fcc3d90
extract public and private keys fields from peerid (#44)
* extract public and private keys fields from peerid

* allow assigning a public key

* cleaned up TODOs

* make pubsub prefix a const

* public key should be an `Option`
2019-12-07 10:36:39 -06:00
Dmitriy Ryajov e623e70e7b
PubSub (Gossip & Flood) Implementation (#36)
This adds gossipsub and floodsub, as well as basic interop testing with the go libp2p daemon. 

* add close event

* wip: gossipsub

* splitting rpc message

* making message handling more consistent

* initial gossipsub implementation

* feat: nim 1.0 cleanup

* wip: gossipsub protobuf

* adding encoding/decoding of gossipsub messages

* add disconnect handler

* add proper gossipsub msg handling

* misc: cleanup for nim 1.0

* splitting floodsub and gossipsub tests

* feat: add mesh rebalansing

* test pubsub

* add mesh rebalansing tests

* testing mesh maintenance

* finishing mcache implementatin

* wip: commenting out broken tests

* wip: don't run heartbeat for now

* switchout debug for trace logging

* testing gossip peer selection algorithm

* test stream piping

* more work around message amplification

* get the peerid from message

* use timed cache as backing store

* allow setting timeout in constructor

* several changes to improve performance

* more through testing of msg amplification

* prevent gc issues

* allow piping to self and prevent deadlocks

* improove floodsub

* allow running hook on cache eviction

* prevent race conditions

* prevent race conditions and improove tests

* use hashes as cache keys

* removing useless file

* don't create a new seq

* re-enable pubsub tests

* fix imports

* reduce number of runs to speed up tests

* break out control message processing

* normalize sleeps between steps

* implement proper transport filtering

* initial interop testing

* clean up floodsub publish logic

* allow dialing without a protocol

* adding multiple reads/writes

* use protobuf varint in mplex

* don't loose conn's peerInfo

* initial interop pubsub tests

* don't duplicate connections/peers

* bring back interop tests

* wip: interop

* re-enable interop and daemon tests

* add multiple read write tests from handlers

* don't cleanup channel prematurely

* use correct channel to send/receive msgs

* adjust tests with latest changes

* include interop tests

* remove temp logging output

* fix ci

* use correct public key serialization

* additional tests for pubsub interop
2019-12-05 20:16:18 -06:00
Dmitriy Ryajov 1d4b51413e option to allow triggering own handlers on publish 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 3b9d34116d decrease floodsub traffic 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 34d1a641de cleanup/test pubsub 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 9862064234 changed copyright year 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 5b3f93ba1c feat: allow multiple handlers per topic in pubsub 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 8920cd7d60 misc: pubsub/floodsub and logging 2019-10-11 08:15:24 +09:00
Dmitriy Ryajov 177eb71ffa wip: floodsub initial implementation 2019-10-11 08:15:24 +09:00