1211 Commits

Author SHA1 Message Date
Tanguy
846baf3853
Various cleanups part 1 ()
* raise -> raise exc
* replace stdlib random with bearssl
* object init -> new
* Remove deprecated procs
* getMandatoryField
2021-10-25 10:26:32 +02:00
Tanguy
3669b90ceb
Fix WS observed address ()
* Fix WS observed address

* Unify tcptransport & wstransport
2021-10-14 13:16:34 +02:00
Tanguy
75bfc1b5f7
Fix muxer bandwidth typo () 2021-09-27 14:30:22 +02:00
Tanguy
1b2cdd6aec
Merge branch 'master' into unstable 2021-09-09 13:22:45 +02:00
Tanguy
68af8122e9
Fix macos go-daemon CI ()
bump libp2p-go-daemon
2021-09-08 16:58:44 +02:00
Menduist
d02735dc46
Remove peer info ()
Peer Info is now for local peer data only.
For other peers info, use the peer store.

Previous reference to peer info are replaced with the peerid
2021-09-08 11:07:46 +02:00
Tanguy Cizain
8cddfde837
Rename getKey -> getPublicKey ()
* rename getKey to getPublicKey

* use publicKey directly in gossipsub

* update error messages
2021-09-02 12:03:40 +02:00
Tanguy Cizain
8ea90037e5
Fix gossipsub incoming graft backoff ()
* Fix gossipsub incoming graft backoff

* Improve debug messages

* clamp to 24h
2021-09-01 08:41:11 +02:00
Tanguy Cizain
62ca6dd9a0
Add missing test to testnative ()
* add missing tests

* fix testping
2021-09-01 08:38:24 +02:00
Tanguy Cizain
f274bfe19d
DNS Addresses handling ()
* add 'dns' multiaddr protocol

* multiaddr: isWire is true for DNS protocols

* resolve dns on connect

* fix typo

* add dns test

* update resolveDns error handling

* handle multiple dns entries

* start of new resolver

* working dns resolver

* use the DnsResolver

* fix json logs

* small overhaul

* fix dns implem in lp2p

* update dnsclient repo

* add dns test to testnative

* dummy dns server for ut

* better mocked

* moved resolving to transport

* moved mockresolver to libp2p

* test resolve in switch test

* try multiple txt & track leaks

* raise e

* catchable error instead of exception

* save failed dns server

* moved resolve back to dialer

* remove nameresolver from dialer
2021-08-18 09:40:12 +02:00
Tanguy Cizain
e58658d822
Fix unstable autobump () 2021-08-03 11:43:00 -06:00
Tanguy Cizain
af3be7966b
Websocket Transport ()
* start of websocket transport

* more ws tests

* switch to common test

* add close to wsstream

* update ws & chronicles version

* cleanup

* removed multicodec

* clean ws outgoing connections

* renamed to websock

* removed stream from logs

* renamed ws to websock

* add connection closing test to common transport

* close incoming connection on ws stop

* renamed testwebsocket.nim -> testwstransport.nim

* removed raise todo

* split out/in connections

* add wss to tests

* Fix tls ()

* change log level

* fixed issue related to stopping

some cosmetic cleanup

* use `allFutures` to stop/close things

Prevent potential race conditions when stopping two or more transports

* misc

* point websock to server-case-object branch

* interop test with go

* removed websock version specification

* add daemon -> native ws test

* fix & test closed read/write

* update readOnce, thanks jangko

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2021-08-03 15:48:03 +02:00
markspanbroek
8b438142ce
Fixes failing peer store test in Nim 1.4.8 ()
* Fixes failing test in Nim 1.4.8

This tests failed because the order in which the elements
of a hashset are added to a seq is non-deterministic.

Co-authored-by: Tanguy Cizain <tanguycizain@gmail.com>
2021-07-28 12:06:18 +02:00
Tanguy Cizain
7fc0dfbd55
Muxer bandwidth metrics ()
* add stream bytes metrics

* renamed bw metric

* renamed

* new global metric
2021-07-26 16:12:36 +02:00
Tanguy Cizain
cb94baf9c4
Fix transport's switch start ()
* fix switch start

* don't fail eagerly on transport start

* handlecancel

* raise exc
2021-07-19 12:51:27 +02:00
Tanguy Cizain
c1b2d45d1b
Track ChronosStream in test ()
* track chronos stream in tests

* use chronosstreamtrackername
2021-07-13 13:53:08 +02:00
Tanguy Cizain
008dcf7043
Faster CI ()
* ci on PR only

* add option to run ci manually

* thanks Stefan
2021-07-13 10:26:49 +02:00
Jacek Sieka
1d4f7c7a43
avoid borrow, breaks logging () 2021-07-11 09:59:42 +02:00
Miran
c12f00c8b6
make isLiteral compatible with ARC and ORC ()
(cherry picked from commit 2d4dad07165cd53e3e81af0dffb462609a93e555)
2021-07-01 12:05:50 +02:00
Tanguy Cizain
26e47d7da5
Various transports improvement ()
* little transport cleanup

* rename TcpTransport.init -> TcpTransport.new

* moved transport e2e to common file

* remove localAddress

* rename testtransport -> testtcptransport

* add checktrackers to commontransports

* removed multicodec from transports
2021-06-30 10:59:30 +02:00
Dmitriy Ryajov
49f137049f
Merge remote-tracking branch 'origin' into unstable 2021-06-29 15:43:11 -06:00
Dmitriy Ryajov
1681197d67
Merge branch 'master' into unstable 2021-06-29 15:42:22 -06:00
Dmitriy Ryajov
398b3a4ce6
Merge branch 'master' into unstable 2021-06-29 11:41:34 -06:00
Ben Tilford
72ccd1d1f5
Correcting paths and file/directory names () 2021-06-27 22:36:57 -06:00
Giovanni Petrantoni
4be0cdcdb4
Docs cleanup ()
* remove pointless go_daemon page (made no sense)

* remove confusing go requirement from main readme

* wip directchat fixes

* use matchPartial for connect(wire)

* fix directchat

* revert wire changes

* rewrite directchat partially

* more readme updates

* fix things to follow the update on blog posts
2021-06-27 20:37:00 -06:00
Tanguy Cizain
7e3974f9c8
Fix flaky macos switch test ()
* sleep to avoid flaky switch test on macos

* Update tests/testswitch.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>

Co-authored-by: markspanbroek <mark@spanbroek.net>
2021-06-23 09:27:52 -06:00
Tanguy Cizain
bee91538ef
builder: add multiple local address support () 2021-06-21 17:14:24 +02:00
Dmitriy Ryajov
b63e064b4a
Remove asynccheck ()
* Merge master ()

* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* Merge master ()

* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* cleanup

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-14 17:21:44 -06:00
Tanguy Cizain
bed00ec43c
Identify Push ()
* start of identifypush

* better pushidentify

* push identify test

* fix: make peerid optional
2021-06-14 11:08:47 -06:00
Tanguy Cizain
bd2e9a0462
Fix trackers tests ()
* Fix checkTracker typo

* fix testswich tests

* use tracker consts
2021-06-14 10:26:11 +02:00
Dmitriy Ryajov
c7d9770637
Merge master ()
* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-11 16:34:41 -06:00
Dmitriy Ryajov
9e2b464e14
Cleanup testinterop imports ()
* cleanup exports/imports, specially for testinterop

* export pubsub
2021-06-11 16:34:40 -06:00
Dmitriy Ryajov
40e21e3375
Cleanup testinterop imports ()
* cleanup exports/imports, specially for testinterop

* export pubsub
2021-06-11 16:33:47 -06:00
Tanguy Cizain
93156447ba
Peer Store implement part II ()
* Connect & Peer event handlers now receive a peerinfo

* small peerstore refacto

* implement peerstore in switch

* changed PeerStore to final ref object

* revert libp2p/builders.nim
2021-06-08 18:55:24 +02:00
Tanguy Cizain
55a3606ecb
add ping protocol ()
* add ping protocol

* add ping protocol handler

* ping styling

* more ping tests

* switch ping to bearssl rng

* update ping style

* new cancellation test
2021-06-08 18:53:45 +02:00
Tanguy Cizain
caac8191d2
Change newXXXX procs to XXXX.new ()
* newBufferStream -> BufferStream.new

* newMultistream -> MultistreamSelect.new

* newSecio -> Secio.new

* newNoise -> Noise.new

* newPlainText -> PlainText.new

* newPubSubPeer -> PubSubPeer.new

* newIdentify -> Identify.new

* newMuxerProvider -> MuxerProvider.new
2021-06-07 09:32:08 +02:00
Ștefan Talpalaru
fbe888a3aa
CI: refactor Nim compiler building () 2021-06-02 12:26:03 -06:00
Dmitriy Ryajov
ac47964377
use pkg namespace 2021-06-02 12:26:02 -06:00
Dmitriy Ryajov
b56ca11b74
use raises defect 2021-06-02 12:26:02 -06:00
Dmitriy Ryajov
ce42674d80
avoid memory safety errors with nim 1.4.x 2021-06-02 12:26:01 -06:00
Dmitriy Ryajov
a6eea0c275
import chronicles otherwise compile breaks 2021-06-02 12:25:37 -06:00
Dmitriy Ryajov
1c3616e3a5
merge latest master 2021-06-02 12:25:36 -06:00
Dmitriy Ryajov
c949f14a99
Merge master to unstable ()
* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* use a builder pattern to build the switch ()

* use a builder pattern to build the switch

* with with

* more refs

* builders ()

* More builders ()

* address some issues pointed out in review

* re-add to prevent breaking other projects

* mem usage cleanups for pubsub ()

In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.

In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.

To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)

* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2

* More gossip coverage ()

* add floodPublish test

* test delivery via control Iwant/have mechanics

* fix issues in control, and add testing

* fix possible backoff issue with pruned routine overriding it

* fix control messages ()

* remove unused control graft check in handleControl

* avoid sending empty Iwant messages

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* Merge master ()

* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* Builders ()

* use a builder pattern to build the switch ()

* use a builder pattern to build the switch

* with with

* more refs

* Merge master ()

* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

* fix builders

* more builder stuff

* more builders

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-02 12:24:46 -06:00
Dmitriy Ryajov
5ee67f31bf
Merge master ()
* builders ()

* More builders ()

* address some issues pointed out in review

* re-add to prevent breaking other projects

* Merge master ()

* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-02 12:24:46 -06:00
Dmitriy Ryajov
530e589e14
Builders ()
* use a builder pattern to build the switch ()

* use a builder pattern to build the switch

* with with

* more refs

* Merge master ()

* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

* fix builders

* more builder stuff

* more builders

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-02 12:24:44 -06:00
Dmitriy Ryajov
ee49b76478
Merge master ()
* Revisit Floodsub ()

Fixes 

add coverage to unsubscribeAll and testing

* add mounted protos to identify message ()

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump ()

* Fix minor exception issues ()

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer ()

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-02 12:24:07 -06:00
Dmitriy Ryajov
eef5dd0042
fix imports 2021-06-02 12:24:06 -06:00
Dmitriy Ryajov
8e3ef540ea
add base exception class and fix hierarchy 2021-06-02 12:24:04 -06:00
Dmitriy Ryajov
a3c00af945
Split dialer ()
* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments
2021-06-02 12:23:44 -06:00
Dmitriy Ryajov
3da656687b
use LPError more consistently ()
* use LPError more consistently

* don't use Exceptino

* annotate with raises

* don't panic on concatenation

* further rework error handling
2021-06-02 15:39:10 +02:00