Commit Graph

164 Commits

Author SHA1 Message Date
lchenut 34c2fb8787
Circuit relay v2 (#717) 2022-08-01 14:31:22 +02:00
lchenut 912873f8b3
Replace table by seq for storing muxers (#741)
Replace table by seq for stocking muxers
2022-07-22 12:54:09 +02:00
Tanguy d0523fdc9d
Documentation (#716) 2022-07-01 20:19:57 +02:00
lchenut 0ece5eaf12
Start/stop protocols (#730)
Starting/stopping a switch now starts/stops all protocols mounted on that switch
2022-06-30 11:21:33 +02:00
Tanguy 60becadcf9
Peer store refacto (#700)
There is now a global PeerStore structure (instead of having one for libp2p, one for waku, etc)

The user can create custom books for new types easily

Also add a pruning system to remove dead peers
2022-05-25 12:12:57 +02:00
lchenut 13503f3799
Circuit relay v1 (#670)
Circuit relay v1
2022-05-18 10:19:37 +02:00
Tanguy 9ba5c069c8
Update SPR when the PeerInfo changes (#711) 2022-04-13 09:35:28 +02:00
Tanguy c09d032133
Allow force dial (#696) 2022-02-24 17:31:47 +01:00
Tanguy df566e69db
Fixes for style check (#676) 2021-12-16 11:05:20 +01:00
Eric Mastro fffa7e8cc2
fix: remove returned Futures from switch.start (#662)
* fix: remove returned Futures from switch.start

The proc `start` returned a seq of futures that was mean to be awaited by the caller. However, the start proc itself awaited each Future before returning it, so the ceremony requiring the caller to await the Future, and returning the Futures themselves was just used to handle errors. But we'll give a better way to handle errors in a future revision

Remove `switch.start` return type (implicit `Future[void]`)

Update tutorials and examples to reflect the change.

* Raise error during failed transport

Replaces logging of error, and adds comment that it should be replaced with a callback in a future PR.
2021-12-03 19:23:12 +01:00
Dmitriy Ryajov 73168b6eae
Add support for multiple addresses to transports (#598)
* add test for multiple local addresses

* allow transports to listen on multiple addrs

* fix tcp transport accept

* check switch addrs are correct

* switch test to port 0

* close accepted peers on close

* ignore CancelledError in transport accept

* test ci

* only accept in accept loop

* avoid accept greedyness

* close acceptedPeers

* accept doesn't crash on cancelled fut

* add common transport test

* close conn on handling failure

* close accepted peers in two steps

* test for macos

* revert accept greedyness

* fix dialing cancel

* test chronos fix

* add ws

* ws cancellation

* small fix

* remove chronos blocked test

* fix testping

* Fix transport's switch start (like #609)

* bump chronos

* Websocket: handle both ws & wss

Co-authored-by: Tanguy Cizain <tanguycizain@gmail.com>
Co-authored-by: Tanguy <tanguy@status.im>
2021-11-24 14:01:12 -06:00
Tanguy c92125a1a4
Integrate dns resolving (#615)
* integrate dns

* give hostname to transport dial

* add hostname test

* switched to websock master

* Add dnsaddr dial test w multiple transports
2021-11-08 13:02:03 +01:00
Tanguy 1b2cdd6aec
Merge branch 'master' into unstable 2021-09-09 13:22:45 +02:00
Menduist d02735dc46
Remove peer info (#610)
Peer Info is now for local peer data only.
For other peers info, use the peer store.

Previous reference to peer info are replaced with the peerid
2021-09-08 11:07:46 +02:00
Tanguy Cizain f274bfe19d
DNS Addresses handling (#580)
* add 'dns' multiaddr protocol

* multiaddr: isWire is true for DNS protocols

* resolve dns on connect

* fix typo

* add dns test

* update resolveDns error handling

* handle multiple dns entries

* start of new resolver

* working dns resolver

* use the DnsResolver

* fix json logs

* small overhaul

* fix dns implem in lp2p

* update dnsclient repo

* add dns test to testnative

* dummy dns server for ut

* better mocked

* moved resolving to transport

* moved mockresolver to libp2p

* test resolve in switch test

* try multiple txt & track leaks

* raise e

* catchable error instead of exception

* save failed dns server

* moved resolve back to dialer

* remove nameresolver from dialer
2021-08-18 09:40:12 +02:00
Tanguy Cizain cb94baf9c4
Fix transport's switch start (#609)
* fix switch start

* don't fail eagerly on transport start

* handlecancel

* raise exc
2021-07-19 12:51:27 +02:00
Tanguy Cizain 93156447ba
Peer Store implement part II (#586)
* Connect & Peer event handlers now receive a peerinfo

* small peerstore refacto

* implement peerstore in switch

* changed PeerStore to final ref object

* revert libp2p/builders.nim
2021-06-08 18:55:24 +02:00
Dmitriy Ryajov 1c3616e3a5
merge latest master 2021-06-02 12:25:36 -06:00
Dmitriy Ryajov a3c00af945
Split dialer (#542)
* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments
2021-06-02 12:23:44 -06:00
Dmitriy Ryajov 3da656687b
use LPError more consistently (#582)
* use LPError more consistently

* don't use Exceptino

* annotate with raises

* don't panic on concatenation

* further rework error handling
2021-06-02 15:39:10 +02:00
Dmitriy Ryajov ac4e060e1a
adding raises defect across the codebase (#572)
* adding raises defect across the codebase

* use unittest2

* add windows deps caching

* update mingw link

* die on failed peerinfo initialization

* use result.expect instead of get

* use expect more consistently and rework inits

* use expect more consistently

* throw on missing public key

* remove unused closure annotation

* merge master
2021-05-21 10:27:01 -06:00
Jacek Sieka 54031c9e9b
Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
2021-03-23 07:45:25 +01:00
Dmitriy Ryajov f7a9d83545
add mounted protos to identify message (#546) 2021-03-15 15:29:05 -06:00
Jacek Sieka 70deac9e0d
fix peer score accumulation (#541)
* fix accumulating peer score
* fix missing exception handling
* remove unnecessary initHashSet/initTable calls
* simplify peer stats management
* clean up tests a little
* fix some missing raises annotations
2021-03-09 13:22:52 +01:00
Giovanni Petrantoni 02ad017107
Gossipsub fixes and Initiator flagging fixes (#539)
* properly propagate initiator information for gossipsub

* Fix pubsubpeer lifetime management

* restore old behavior

* tests fixing

* clamp backoff time value received

* fix member name collisions

* internal test fixes

* better names and explaining of the importance of transport direction

* fixes
2021-03-03 08:23:40 +09:00
Ștefan Talpalaru d9563d65ae
support compilation with Nim-1.4 HEAD (#521) 2021-02-08 15:21:43 -06:00
Dmitriy Ryajov 0959877b29
Connection limits (#384)
* master merge

* wip

* avoid deadlocks

* tcp limits

* expose client field in chronosstream

* limit incoming connections

* update with new listen api

* fix release

* don't override peerinfo in connection

* rework transport with accept

* use semaphore to track resource ussage

* rework with new transport accept api

* move events to conn manager (#373)

* use semaphore to track resource ussage

* merge master

* expose api to acquire conn slots

* don't fail expensive metrics

* allow tracking and updating connections

* set global connection limits to 80

* add per peer connection limits

* make sure conn is closed if tracking failed

* more descriptive naming for handle

* rework with new transport accept api

* add `getStream` hide `selectConn`

* add TransportClosedError

* make nil explicit

* don't make unnecessary copies of message

* logging

* error handling

* cleanup semaphore

* track connections properly

* throw `TooManyConnections` when tracking outgoing

* use proper exception and handle conventions

* check onCloseHandle for nil

* revert internalConnect changes

* adding upgraded flag

* await stream before closing

* simplify tracking

* wip

* logging

* split connection limits into incoming and outgoing

* further streamline connection limits split counts

* don't use closeWithEOF

* move peer and conn event triggers from switch

* wip

* wip

* wip

* merge master

* handle nil connections properly

* add clarifying comment

* don't raise exc on nil

* no finally

* add proper min/max connections logic

* rebase master

* merge master

* master merge

* remove request timeout

should be addressed in separate PR

* merge master

* share semaphore when in/out limits arent enforced

* merge master

* use import

* pass semaphore to trackConn

* don't close last conn

* use storeConn

* merge master

* use storeConn
2021-01-20 22:00:24 -06:00
Dmitriy Ryajov 96c01e5e69
Split upgrade flow (#507)
* splitting upgrade flow

* bring back master changes

* re-export `Upgrade`

* export public methods/procs in derived class

* style fixes
2021-01-20 11:28:32 -06:00
Dmitriy Ryajov 34e330353f
better `upgraded` lifetime handling (avoid NPE) (#506)
* avoid npe on connection upgrade

* add `onUpgraded` event
2021-01-18 16:27:29 -06:00
Giovanni Petrantoni 240ec84ffb
Gossipsub wip (#502)
* Remove unused connections in pubsubpeer, also removed wrong usages, add a disconnect bad peers parameter

* handle exceptions in disconnectPeer

* small fix

* use the proper disconnection procedure for gossip peers

* fixes, more metrics add test about disconnection

* hot fix possible null pointers in switch

* silly isnil sugar

* Fix and test gossip directPeer connections
2021-01-15 13:48:03 +09:00
Dmitriy Ryajov 3878a95b23
Semaphore cancellations (#503)
* add proper cancelation handling

* remove cancelled futures explicitly

* use fifo to keep proper order

* add out of order cancelations test

* make count public

* use `new` instead of `init`

* remove private `queue` from tests

* expose count as a readonly prop

* use `delete()` to preserve seq order
2021-01-14 10:11:12 +01:00
Dmitriy Ryajov 8e57746f3a
improving connection estblishing metrics (#500) 2021-01-07 17:06:41 -06:00
Dmitriy Ryajov b2ea5a3c77
Concurrent upgrades (#489)
* adding an upgraded event to conn

* set stopped flag asap

* trigger upgradded event on conn

* set concurrency limit for accepts

* backporting semaphore from tcp-limits2

* export unittests module

* make params explicit

* tone down debug logs

* adding semaphore tests

* use semaphore to throttle concurent upgrades

* add libp2p scope

* trigger upgraded event before any other events

* add event handler for connection upgrade

* cleanup upgraded event on conn close

* make upgrades slot release rebust

* dont forget to release slot on nil connection

* misc

* make sure semaphore is always released

* minor improvements and a nil check

* removing unneeded comment

* make upgradeMonitor a non-closure proc

* make sure the `upgraded` event is initialized

* handle exceptions in accepts when stopping

* don't leak exceptions when stopping accept loops
2021-01-04 12:59:05 -06:00
Dmitriy Ryajov e9d4679059
Race in connection setup (#464)
* check that connection is not closed or eof

* don't release connection lock prematurely

* test that only valid connections can be added

* correct exception type on closed connection

* add clarifying comment

* use closeWithEOF for more stable test

* misc comments

* log stream id in buffestream asserts

* use closeWithEOF to prevent races in tests

* give some time to the remote handler to trigger

* adding more tests to make codecov happy
2020-12-02 19:24:48 -06:00
Dmitriy Ryajov d1c689e5ab
adding libp2p tag to logScope (#465) 2020-12-01 11:34:27 -06:00
Dmitriy Ryajov 18443dafc1
rework peer event to take an initiator flag (#456)
* rework peer event to take an initiator flag

* use correct direction for initiator
2020-11-28 10:59:47 -06:00
Dmitriy Ryajov a8f5f7a8bb
move dialing logic to it's own proc to avoid try/finally bugs (#461)
* move dialing logic to it's own proc to avoid try/finally bugs

* re-export transport

* lint

* add cancelation test

* test remote conn close on dial
2020-11-28 09:05:12 +01:00
Dmitriy Ryajov 351489bfa9
getMuxedStream to more appropriate getStream (#448) 2020-11-24 00:37:45 -06:00
Dmitriy Ryajov 92fa4110c1
Rework transport to use chronos accept (#420)
* rework transport to use the new accept api

* use the new chronos primits

* fixup tests to use the new transport api

* handle all exceptions in upgradeIncoming

* master merge

* add multiaddress exception type

* raise appropriate exception on invalida address

* allow retrying on TransportTooManyError

* adding TODO

* wip

* merge master

* add sleep if nil is returned

* accept loop handles all exceptions

* avoid issues with tray/except/finally

* make consistent with master

* cleanup accept loop

* logging

* Update libp2p/transports/tcptransport.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

* use Direction enum instead of initiator flag

* use consistent import style

* remove experimental `closeWithEOF()`

Co-authored-by: Jacek Sieka <jacek@status.im>
2020-11-18 20:06:42 -06:00
Giovanni Petrantoni 7cc42ce219
start adding more tests + minor fixes (#419)
* start adding more tests + minor fixes

* add wrong secure negotiation test

* add noise failed handshake test
2020-11-04 23:24:41 +09:00
Dmitriy Ryajov 7b5259dbc7
Move triggers (#416)
* move event triggers to connmanager

* use base error type

* avoid deadlocks

* handle eof and closed when identifying incoming

* use `closeWait`
2020-11-02 14:35:26 -06:00
Jacek Sieka 03639f1446
Revert "Channel leaks (#413)" (#417)
This reverts commit 1de1d49223.
2020-11-01 14:49:25 -06:00
Dmitriy Ryajov 1de1d49223
Channel leaks (#413)
* break stream tracking by type

* use closeWithEOF to await wrapped stream

* fix cancelation leaks

* fix channel leaks

* logging

* use close monitor and always call closeUnderlying

* don't use closeWithEOF

* removing close monitor

* logging
2020-10-27 11:21:03 -06:00
Dean Eigenmann 853238a215
feature/expose-matcher (#387)
* exposes matcher

* might work

* fix

* fix
2020-10-02 08:59:15 -06:00
Jacek Sieka 25bd0a18f4
small fixes (#374)
* add helper to read EOF marker after closing stream (else stream stay
alive until timeout/reset)
* don't assert on empty channel message
* don't loop when writing to chronos (no need)
2020-09-24 07:30:19 +02:00
Dmitriy Ryajov abd234601b
move events to conn manager (#373) 2020-09-23 08:07:16 -06:00
Giovanni Petrantoni b99d2039a8
Gossip one one (#240)
* allow multiple codecs per protocol (without breaking things)

* add 1.1 protocol to gossip

* explicit peering part 1

* explicit peering part 2

* explicit peering part 3

* PeerInfo and ControlPrune protocols

* fix encodePrune

* validated always, even explicit peers

* prune by score (score is stub still)

* add a way to pass parameters to gossip

* standard setup fixes

* take into account explicit direct peers in publish

* add floodPublish logic

* small fixes, publish still half broken

* make sure to waitsub in sparse test

* use var semantics to optimize table access

* wip... lvalues don't work properly sadly...

* big publish refactor, replenish and balance

* fix internal tests

* use g.peers for fanout (todo: don't include flood peers)

* exclude non gossip from fanout

* internal test fixes

* fix flood tests

* fix test's trypublish

* test interop fixes

* make sure to not remove peers from gossip table

* restore old replenishFanout

* cleanups

* restore utility module import

* restore trace vs debug in gossip

* improve fanout replenish behavior further

* triage publish nil peers (issue is on master too but just hidden behind a if/in)

* getGossipPeers fixes

* remove topics from pubsubpeer (was unused)

* simplify rebalanceMesh (following spec) and make it finally reach D_high

* better diagnostics

* merge new pubsubpeer, copy 1.1 to new module

* fix up merge

* conditional enable gossip11 module

* add back topics in peers, re-enable flood publish

* add more heartbeat locking to prevent races

* actually lock the heartbeat

* minor fixes

* with sugar

* merge 1.0

* remove assertion in publish

* fix multistream 1.1 multi proto

* Fix merge oops

* wip

* fix gossip 11 upstream

* gossipsub11 -> gossipsub

* support interop testing

* tests fixing

* fix directchat build

* control prune updates (pb)

* wip parameters

* gossip internal tests fixes

* parameters wip

* finishup with params

* cleanups/wip

* small sugar

* grafted and pruned procs

* wip updateScores

* wip

* fix logging issue

* pubsubpeer, chronicles explicit override

* fix internal gossip tests

* wip

* tables troubleshooting

* score wip

* score wip

* fixes

* fix test utils generateNodes

* don't delete while iterating in score update

* fix grafted defect

* add a handleConnect in subscribeTopic

* pruning improvements

* wip

* score fixes

* post merge - builds gossip tests

* further merge fixes

* rebalance improvements and opportunistic grafting

* fix test for now

* restore explicit peering

* implement peer exchange graft message

* add an hard cap to PX

* backoff time management

* IWANT cap/budget

* Adaptive gossip dissemination

* outbound mesh quota, internal tests fixing

* oversub prune score based, finish outbound quota

* finishup with score and ihave budget

* use go daemon 0.3.0

* import fixes

* byScore cleanup score sorting

* remove pointless scaling in `/` Duration operator

* revert using libp2p org for daemon

* interop fixes

* fixes and cleanup

* remove heartbeat assertion, minor debug fixes

* logging improvements and cleaning up

* (to revert) add some traces

* add explicit topic to gossip rpcs

* pubsub merge fixes and type fix in switch

* Revert "(to revert) add some traces"

This reverts commit 4663eaab6cc336c81cee50bc54025cf0b7bcbd99.

* cleanup some now irrelevant todo

* shuffle peers anyway as score might be disabled

* add missing shuffle

* old merge fix

* more merge fixes

* debug improvements

* re-enable gossip internal tests

* add gossip10 fallback (dormant but tested)

* split gossipsub internal tests into 1.0 and 1.1

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2020-09-21 11:16:29 +02:00
Dmitriy Ryajov b0d86b95dd
add peer lifecycle events (#357)
* add peer lifecycle events

* rework peer events to not use connection events

* don't use result in pubsub and switch init

* wip

* use ordered hashes and remove logscope

* logging

* add missing test

* small fixes
2020-09-15 14:19:22 -06:00
Jacek Sieka 5b347adf58
logging fixes and small cleanups (#361)
In particular, allow longer multistream select reads
2020-09-09 19:12:08 +02:00
Jacek Sieka 2b72d485a3
a few more log fixes (#355) 2020-09-07 14:15:11 +02:00