Commit Graph

48 Commits

Author SHA1 Message Date
lchenut 13503f3799
Circuit relay v1 (#670)
Circuit relay v1
2022-05-18 10:19:37 +02:00
Eric Mastro fffa7e8cc2
fix: remove returned Futures from switch.start (#662)
* fix: remove returned Futures from switch.start

The proc `start` returned a seq of futures that was mean to be awaited by the caller. However, the start proc itself awaited each Future before returning it, so the ceremony requiring the caller to await the Future, and returning the Futures themselves was just used to handle errors. But we'll give a better way to handle errors in a future revision

Remove `switch.start` return type (implicit `Future[void]`)

Update tutorials and examples to reflect the change.

* Raise error during failed transport

Replaces logging of error, and adds comment that it should be replaced with a callback in a future PR.
2021-12-03 19:23:12 +01:00
Menduist d02735dc46
Remove peer info (#610)
Peer Info is now for local peer data only.
For other peers info, use the peer store.

Previous reference to peer info are replaced with the peerid
2021-09-08 11:07:46 +02:00
Tanguy Cizain af3be7966b
Websocket Transport (#593)
* start of websocket transport

* more ws tests

* switch to common test

* add close to wsstream

* update ws & chronicles version

* cleanup

* removed multicodec

* clean ws outgoing connections

* renamed to websock

* removed stream from logs

* renamed ws to websock

* add connection closing test to common transport

* close incoming connection on ws stop

* renamed testwebsocket.nim -> testwstransport.nim

* removed raise todo

* split out/in connections

* add wss to tests

* Fix tls (#608)

* change log level

* fixed issue related to stopping

some cosmetic cleanup

* use `allFutures` to stop/close things

Prevent potential race conditions when stopping two or more transports

* misc

* point websock to server-case-object branch

* interop test with go

* removed websock version specification

* add daemon -> native ws test

* fix & test closed read/write

* update readOnce, thanks jangko

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2021-08-03 15:48:03 +02:00
Dmitriy Ryajov 9e2b464e14
Cleanup testinterop imports (#589)
* cleanup exports/imports, specially for testinterop

* export pubsub
2021-06-11 16:34:40 -06:00
Dmitriy Ryajov ac4e060e1a
adding raises defect across the codebase (#572)
* adding raises defect across the codebase

* use unittest2

* add windows deps caching

* update mingw link

* die on failed peerinfo initialization

* use result.expect instead of get

* use expect more consistently and rework inits

* use expect more consistently

* throw on missing public key

* remove unused closure annotation

* merge master
2021-05-21 10:27:01 -06:00
Giovanni Petrantoni 795a651839
use a builder pattern to build the switch (#551)
* use a builder pattern to build the switch

* with with

* more refs
2021-04-02 10:20:51 +09:00
Dmitriy Ryajov 4dea23c394
Remove secio usage and cleanup exports (#519)
* cleaned up exports

* remove secio use

* added more useful exports

* proper import
2021-02-08 14:33:34 -06:00
Giovanni Petrantoni 05e789a34f
Gossipsub refactor (#490)
* refactor peerStats, re-enable scores for testing

* remove gossip 1.0

* cleanup

* codecov matrix fixes

* restore previous score on onNewPeer

* fix coverage n checks

* unsubscribeAll gossipsub fixes

* refactor unsub/sub

* refactor onNewPeer and fix score flow

* disable scores by default (change in tests later)

* fix tests, enable scores in tests

* fix wrongly merged test

* ensure topic removal from topics table

* small typo fix

* testinterop fixes
2020-12-19 15:43:32 +01:00
Dmitriy Ryajov 55b763264e
Cleanup tests (#435)
* add async testing methods

* refactor with async testing methods

* use iffy in async tests
2020-11-12 21:44:02 -06:00
Giovanni Petrantoni bd70515087
fix first interop test (#398) 2020-10-15 10:37:20 +09:00
Jacek Sieka 58f26d4164
temporarily skip broken test on windows (#375) 2020-09-24 14:50:04 +02:00
Giovanni Petrantoni b99d2039a8
Gossip one one (#240)
* allow multiple codecs per protocol (without breaking things)

* add 1.1 protocol to gossip

* explicit peering part 1

* explicit peering part 2

* explicit peering part 3

* PeerInfo and ControlPrune protocols

* fix encodePrune

* validated always, even explicit peers

* prune by score (score is stub still)

* add a way to pass parameters to gossip

* standard setup fixes

* take into account explicit direct peers in publish

* add floodPublish logic

* small fixes, publish still half broken

* make sure to waitsub in sparse test

* use var semantics to optimize table access

* wip... lvalues don't work properly sadly...

* big publish refactor, replenish and balance

* fix internal tests

* use g.peers for fanout (todo: don't include flood peers)

* exclude non gossip from fanout

* internal test fixes

* fix flood tests

* fix test's trypublish

* test interop fixes

* make sure to not remove peers from gossip table

* restore old replenishFanout

* cleanups

* restore utility module import

* restore trace vs debug in gossip

* improve fanout replenish behavior further

* triage publish nil peers (issue is on master too but just hidden behind a if/in)

* getGossipPeers fixes

* remove topics from pubsubpeer (was unused)

* simplify rebalanceMesh (following spec) and make it finally reach D_high

* better diagnostics

* merge new pubsubpeer, copy 1.1 to new module

* fix up merge

* conditional enable gossip11 module

* add back topics in peers, re-enable flood publish

* add more heartbeat locking to prevent races

* actually lock the heartbeat

* minor fixes

* with sugar

* merge 1.0

* remove assertion in publish

* fix multistream 1.1 multi proto

* Fix merge oops

* wip

* fix gossip 11 upstream

* gossipsub11 -> gossipsub

* support interop testing

* tests fixing

* fix directchat build

* control prune updates (pb)

* wip parameters

* gossip internal tests fixes

* parameters wip

* finishup with params

* cleanups/wip

* small sugar

* grafted and pruned procs

* wip updateScores

* wip

* fix logging issue

* pubsubpeer, chronicles explicit override

* fix internal gossip tests

* wip

* tables troubleshooting

* score wip

* score wip

* fixes

* fix test utils generateNodes

* don't delete while iterating in score update

* fix grafted defect

* add a handleConnect in subscribeTopic

* pruning improvements

* wip

* score fixes

* post merge - builds gossip tests

* further merge fixes

* rebalance improvements and opportunistic grafting

* fix test for now

* restore explicit peering

* implement peer exchange graft message

* add an hard cap to PX

* backoff time management

* IWANT cap/budget

* Adaptive gossip dissemination

* outbound mesh quota, internal tests fixing

* oversub prune score based, finish outbound quota

* finishup with score and ihave budget

* use go daemon 0.3.0

* import fixes

* byScore cleanup score sorting

* remove pointless scaling in `/` Duration operator

* revert using libp2p org for daemon

* interop fixes

* fixes and cleanup

* remove heartbeat assertion, minor debug fixes

* logging improvements and cleaning up

* (to revert) add some traces

* add explicit topic to gossip rpcs

* pubsub merge fixes and type fix in switch

* Revert "(to revert) add some traces"

This reverts commit 4663eaab6c.

* cleanup some now irrelevant todo

* shuffle peers anyway as score might be disabled

* add missing shuffle

* old merge fix

* more merge fixes

* debug improvements

* re-enable gossip internal tests

* add gossip10 fallback (dormant but tested)

* split gossipsub internal tests into 1.0 and 1.1

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-09-21 11:16:29 +02:00
Dmitriy Ryajov b0d86b95dd
add peer lifecycle events (#357)
* add peer lifecycle events

* rework peer events to not use connection events

* don't use result in pubsub and switch init

* wip

* use ordered hashes and remove logscope

* logging

* add missing test

* small fixes
2020-09-15 14:19:22 -06:00
Jacek Sieka 0db45462cd
mplex fixes (#362)
* remove almost-empty types module
* lock when writing message (that's the only place the lock matters, and
only when the message is > max msg size)
* logging updates (log in consistent order, makes reading logs easier)
* raise EOF from readExactly only if no bytes have been read (to signal
that _no_ bytes were lost)
2020-09-14 10:19:54 +02:00
Dmitriy Ryajov b76b3e0e9b
Rework pubsub (#322)
* move pubsub of off switch, pass switch into pubsub

* use join on lpstreams

* properly cleanup up failed peers

* fix tests

* fix peertable hasPeerId

* fix tests

* rework sending, remove helpers from pubsubpeer, unify in broadcast

* further split broadcast into send

* use send where appropriate

* use formatIt

* improve trace

Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-08-11 18:05:49 -06:00
Dmitriy Ryajov f7fdf31365
Pubsub lifetime (#284)
* lifecycle hooks

* tests

* move trace after closed check

* restore 1 second heartbeat

* await close event

* fix tests

* print direction string

* more trace logging

* add pubsub monitor

* add log scope

* adjust idle timeout

* add exc.msg to trace
2020-07-27 13:33:51 -06:00
Dmitriy Ryajov ba071cafa6
Channel timeout (#278)
* add support for channel timeouts

* tests for channel timeout

* add timeouts to standard switch

* fix mplex init

* cleanup timer on stream close

* add comment for `isConnected`

* move cleanup event
2020-07-17 12:44:41 -06:00
Dmitriy Ryajov a52763cc6d
fix publishing (#250)
* use var semantics to optimize table access

* wip... lvalues don't work properly sadly...

* big publish refactor, replenish and balance

* fix internal tests

* use g.peers for fanout (todo: don't include flood peers)

* exclude non gossip from fanout

* internal test fixes

* fix flood tests

* fix test's trypublish

* test interop fixes

* make sure to not remove peers from gossip table

* restore old replenishFanout

* cleanups

* Cleanup resources (#246)

* consolidate reading in lpstream

* remove debug echo

* tune log level

* add channel cleanup and cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cancelation handling

* cleanup and cancelation handling

* cancelation handling

* cancelation

* tests

* rename isConnected to connected

* remove testing trace

* comment out debug stacktraces

* explicit raises

* restore trace vs debug in gossip

* improve fanout replenish behavior further

* cleanup stale peers more eaguerly

* synchronize connection cleanup and small refactor

* close client first and call parent second

* disconnect failed peers on publish

* check for publish result

* fix tests

* fix tests

* always call close

Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-07-07 18:33:05 -06:00
Giovanni Petrantoni ec00c7fc50
Peer resultification and defect only (#245)
* Peer resultification and defect only

* Fixing some tests

* test fixes

* Rename peer into peerid

* better result error message in identify

* further merge fixes
2020-07-01 08:25:09 +02:00
Dmitriy Ryajov 7a95f1844b
Concurrent dials (#238)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* add secureconn constructor

* close in the correct order

* concurent dial lock and track in/out conns better

* make tests pass

* add todo comment

* disconect peers that open too many connections

* wip

* do connection and muxer tracking in one place

* prevent nil pointer in observers

* drop connections when peers is over max

* prevent channel leaks

* don't use closure to handle channel
2020-06-24 09:08:44 -06:00
Giovanni Petrantoni 7852c6dd0f
Noise and eth2/nbc fixes (#226)
* Remove noise padding payload (spec removed it)

* add log scope in secure

* avoid defect array out of range in switch secure when "na"

* improve identify traces

* wip noise fixes

* noise protobuf adjustments (trying)

* add more debugging messages/traces, improve their actual contents

* re-enable ID check in noise

* bump go daemon tag version

* bump go daemon tag version

* enable noise in daemonapi

* interop testing, (both secio and noise will be tested)

* azure cache bump (p2pd)

* CI changes

- Travis: use Go 1.14
- azure-pipelines.yml: big cleanup
- Azure: bump cache keys
- build 64-bit p2pd on 32-bit Windows
- install both Mingw-w64 architectures

* noise logging fixes

* alternate testing between noise and secio

* increase timeout to avoid VM errors in CI (multistream tests)

* refactor heartbeat management in gossipsub

* remove locking within heartbeat

* refactor heartbeat management in gossipsub

* remove locking within heartbeat

Co-authored-by: Ștefan Talpalaru <stefantalpalaru@yahoo.com>
2020-06-20 19:56:55 +09:00
Dmitriy Ryajov 5b28e8c488
Cleanup lpstream, Connection and BufferStream (#228)
* count published messages

* don't call `switch.dial` in `subscribeToPeer`

* don't use delegation in connection

* move connection out to own file

* don't breakout on reset

* make sure to call close on secured conn

* add lpstream tracing

* don't breackdown by conn id

* fix import

* remove unused lable

* reset  connection on exception

* add additional metrics for skipped messages

* check for nil in secure.close
2020-06-19 11:29:43 -06:00
Dmitriy Ryajov 5960d42c50
remove casts from (#203) 2020-06-02 20:21:11 -06:00
Dmitriy Ryajov abf659a01a more consistent dialing proto selecting logic 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 4c191866e4 use all() for futures and track connections 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 4df151a3a3 typos 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov daef00fc7b don't crash schlesi-dev 2020-06-02 09:10:27 -06:00
Dmitriy Ryajov 20c68a2018 use all() for futures and track connections 2020-06-02 09:10:27 -06:00
Giovanni Petrantoni 37b98ad45c
Secure managers are now sorted, giving priority to noise (#191)
* Secure managers are now sorted, giving priority to noise

* fix nimble test command

* Fix native tests

* fix directchat sample

* Could not write to connection - reduce verbosity

* fix interop testing

* Remove more tables

* test interop fixes

* directchat fix

* fix interop/remove some deprecation
2020-06-01 08:41:32 +02:00
Giovanni Petrantoni 6affcda937
Less exceptions more results (#188)
* Less exceptions more results

* Fix daemonapi and interop tests

* Add multibase

* wip multiaddress

* fix the build, consuming new result types

* fix standard setup

* Simplify match, rename into MaError, add more exaustive err text

* Fix the CI issues

* Fix directchat build

* daemon api fixes

* better err messages formatting

Co-authored-by: Zahary Karadjov <zahary@gmail.com>
2020-05-31 16:22:49 +02:00
Dmitriy Ryajov 9cf1fd0216
remove generic constructor and expose serverflags (#176)
* remove generic constructor and expose serverflags

* fix transport constructor

* fix merge issues
2020-05-18 13:04:05 -06:00
Dmitriy Ryajov 1819502fb5
Cleanup - tests and logging (#178)
* make async for proper exception handling

* tryAndWarn msg messes up Exception msg

* misc: comment out tracker dumps

* cleanup mplex tests

* more informative errors

* give CI time to run

* revert change, bacause it causes races
2020-05-18 07:49:49 -06:00
Giovanni Petrantoni 7dcb807f64
Crypto utilities resultification (#150) 2020-05-18 07:25:55 +02:00
Jacek Sieka 69abf5097d
handle a few exceptions (#170)
* handle a few exceptions

Some of these are maybe too aggressive, but in return, they'll log
their exception - more refactoring needed to sort this out - right now
we get crashes on unhandled exceptions of unknown origin

* during connection setup
* while closing channels
* while processing pubsubs

* catch exceptions that are raised and don't try to catch exceptions that are not raised

* propagate cancellederror

* one more

* more

* more

* make interop tests less fragile

* Raise expiration time in gossipsub fanout test for slow CI

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Giovanni Petrantoni <giovanni@fragcolor.xyz>
2020-05-14 21:56:56 -06:00
Jacek Sieka 3053f03814 fix varint issues
* fixes #111
2020-05-11 09:12:23 -06:00
Jacek Sieka 1efada474c
remove readLoop in secure protocols (#162)
* remove readLoop in secure protocols, fix security issues

* fix Defect on remote sending 0-byte noise/secio message
* remove msglen from `write` (unused)
* simplify SecureConn data flow
* document some control-flow issues

* unify exception behaviour across noise and secio

* secio would not raise on mac/decryption errors

* fix compile error
2020-05-07 14:37:46 -06:00
Dmitriy Ryajov 5285f0d091
Fix/misc (#116)
* only check for payload size

* only subscribe if connection succeeded

* fix failing test

* check that the strem is active before openning

* msg type should not be > than 0x7

* fix tests

* check max against enum val
2020-03-29 08:28:48 -06:00
Dmitriy Ryajov a92d616a3e fix tests 2020-03-28 09:29:43 -06:00
Eugene Kabanov 5701d937c8
Signed variable integers fixes. (#96)
* Fix signed varints.
Add tests for signed varints.
Remove some casts to allow usage at compile time.

* Fix vsizeof() on 32bit platforms.

* Add `hint` and `zint` types for proper signed integer encoding.

* Fix varint related bugs.

* Update requirements.

* Fix interop tests because of fixed readLine.

* Add putVarint, getVarint and tests.
2020-03-06 20:19:43 +01:00
Dmitriy Ryajov eb49d4b218 no empty proto dials and add connect method 2020-02-25 17:52:08 -06:00
Dmitriy Ryajov fbcef69891 implicitelly dial pubsub if enabled 2020-02-21 09:21:06 -06:00
Dmitriy Ryajov 7f8eb0272e cleanup and fix tests 2020-02-16 11:31:35 -06:00
Eugene Kabanov 540e79a430
Add peer lifetime feature for PeerInfo. (#77)
* Add peer lifetime feature for PeerInfo.
Refactor peerinfo to use openarrays instead of sequences.
Fix tests and examples to use arrays instead of sequences.

* Add access to lifetime Future[T] itself.
2020-02-11 12:53:39 -05:00
Dmitriy Ryajov 0fb1f1c5b8 strenghten pubsub interop testing 2019-12-24 10:35:35 -06:00
Dmitriy Ryajov 293a219dbe
Cleanup (#55)
* fix: don't allow replacing pubkey

* fix: several small improvements

* removing pubkey setter

* improove error handling

* remove the use of Option[T] if not needed

* don't use optional

* fix-ci: temporarily pin p2pd to a working tag

* fix example to comply with latest changes

* bumping p2pd again to a higher version
2019-12-10 14:50:35 -06:00
Dmitriy Ryajov 5f6fcc3d90
extract public and private keys fields from peerid (#44)
* extract public and private keys fields from peerid

* allow assigning a public key

* cleaned up TODOs

* make pubsub prefix a const

* public key should be an `Option`
2019-12-07 10:36:39 -06:00
Dmitriy Ryajov e623e70e7b
PubSub (Gossip & Flood) Implementation (#36)
This adds gossipsub and floodsub, as well as basic interop testing with the go libp2p daemon. 

* add close event

* wip: gossipsub

* splitting rpc message

* making message handling more consistent

* initial gossipsub implementation

* feat: nim 1.0 cleanup

* wip: gossipsub protobuf

* adding encoding/decoding of gossipsub messages

* add disconnect handler

* add proper gossipsub msg handling

* misc: cleanup for nim 1.0

* splitting floodsub and gossipsub tests

* feat: add mesh rebalansing

* test pubsub

* add mesh rebalansing tests

* testing mesh maintenance

* finishing mcache implementatin

* wip: commenting out broken tests

* wip: don't run heartbeat for now

* switchout debug for trace logging

* testing gossip peer selection algorithm

* test stream piping

* more work around message amplification

* get the peerid from message

* use timed cache as backing store

* allow setting timeout in constructor

* several changes to improve performance

* more through testing of msg amplification

* prevent gc issues

* allow piping to self and prevent deadlocks

* improove floodsub

* allow running hook on cache eviction

* prevent race conditions

* prevent race conditions and improove tests

* use hashes as cache keys

* removing useless file

* don't create a new seq

* re-enable pubsub tests

* fix imports

* reduce number of runs to speed up tests

* break out control message processing

* normalize sleeps between steps

* implement proper transport filtering

* initial interop testing

* clean up floodsub publish logic

* allow dialing without a protocol

* adding multiple reads/writes

* use protobuf varint in mplex

* don't loose conn's peerInfo

* initial interop pubsub tests

* don't duplicate connections/peers

* bring back interop tests

* wip: interop

* re-enable interop and daemon tests

* add multiple read write tests from handlers

* don't cleanup channel prematurely

* use correct channel to send/receive msgs

* adjust tests with latest changes

* include interop tests

* remove temp logging output

* fix ci

* use correct public key serialization

* additional tests for pubsub interop
2019-12-05 20:16:18 -06:00