* add 'dns' multiaddr protocol
* multiaddr: isWire is true for DNS protocols
* resolve dns on connect
* fix typo
* add dns test
* update resolveDns error handling
* handle multiple dns entries
* start of new resolver
* working dns resolver
* use the DnsResolver
* fix json logs
* small overhaul
* fix dns implem in lp2p
* update dnsclient repo
* add dns test to testnative
* dummy dns server for ut
* better mocked
* moved resolving to transport
* moved mockresolver to libp2p
* test resolve in switch test
* try multiple txt & track leaks
* raise e
* catchable error instead of exception
* save failed dns server
* moved resolve back to dialer
* remove nameresolver from dialer
* start of websocket transport
* more ws tests
* switch to common test
* add close to wsstream
* update ws & chronicles version
* cleanup
* removed multicodec
* clean ws outgoing connections
* renamed to websock
* removed stream from logs
* renamed ws to websock
* add connection closing test to common transport
* close incoming connection on ws stop
* renamed testwebsocket.nim -> testwstransport.nim
* removed raise todo
* split out/in connections
* add wss to tests
* Fix tls (#608)
* change log level
* fixed issue related to stopping
some cosmetic cleanup
* use `allFutures` to stop/close things
Prevent potential race conditions when stopping two or more transports
* misc
* point websock to server-case-object branch
* interop test with go
* removed websock version specification
* add daemon -> native ws test
* fix & test closed read/write
* update readOnce, thanks jangko
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Fixes failing test in Nim 1.4.8
This tests failed because the order in which the elements
of a hashset are added to a seq is non-deterministic.
Co-authored-by: Tanguy Cizain <tanguycizain@gmail.com>
* little transport cleanup
* rename TcpTransport.init -> TcpTransport.new
* moved transport e2e to common file
* remove localAddress
* rename testtransport -> testtcptransport
* add checktrackers to commontransports
* removed multicodec from transports
* remove pointless go_daemon page (made no sense)
* remove confusing go requirement from main readme
* wip directchat fixes
* use matchPartial for connect(wire)
* fix directchat
* revert wire changes
* rewrite directchat partially
* more readme updates
* fix things to follow the update on blog posts
* Merge master (#555)
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* Merge master (#555)
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* cleanup
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* Connect & Peer event handlers now receive a peerinfo
* small peerstore refacto
* implement peerstore in switch
* changed PeerStore to final ref object
* revert libp2p/builders.nim
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* use a builder pattern to build the switch (#551)
* use a builder pattern to build the switch
* with with
* more refs
* builders (#559)
* More builders (#560)
* address some issues pointed out in review
* re-add to prevent breaking other projects
* mem usage cleanups for pubsub (#564)
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.
In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.
To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)
* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
* More gossip coverage (#553)
* add floodPublish test
* test delivery via control Iwant/have mechanics
* fix issues in control, and add testing
* fix possible backoff issue with pruned routine overriding it
* fix control messages (#566)
* remove unused control graft check in handleControl
* avoid sending empty Iwant messages
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* Merge master (#555)
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* Builders (#558)
* use a builder pattern to build the switch (#551)
* use a builder pattern to build the switch
* with with
* more refs
* Merge master (#555)
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
* fix builders
* more builder stuff
* more builders
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* builders (#559)
* More builders (#560)
* address some issues pointed out in review
* re-add to prevent breaking other projects
* Merge master (#555)
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* use a builder pattern to build the switch (#551)
* use a builder pattern to build the switch
* with with
* more refs
* Merge master (#555)
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
* fix builders
* more builder stuff
* more builders
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* Revisit Floodsub (#543)
Fixes#525
add coverage to unsubscribeAll and testing
* add mounted protos to identify message (#546)
* add stable/unstable auto bumps
* fix auto-bump CI
* merge nbc auto bump with CI in order to bump only on CI success
* put conditional locks on nbc bump (#549)
* Fix minor exception issues (#550)
Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.
* fix nimbus ref for auto-bump stable's PR
* Split dialer (#542)
* extracting dialing logic to dialer
* exposing upgrade methods on transport
* cleanup
* fixing tests to use new interfaces
* add comments
* add base exception class and fix hierarchy
* fix imports
* `doAssert` is `ValueError` not `AssertionError`?
* revert back to `AssertionError`
Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
* adding raises defect across the codebase
* use unittest2
* add windows deps caching
* update mingw link
* die on failed peerinfo initialization
* use result.expect instead of get
* use expect more consistently and rework inits
* use expect more consistently
* throw on missing public key
* remove unused closure annotation
* merge master
* gossipsub: unsubscribe fixes
* fix KeyError when updating metric of unsubscribed topic
* fix unsubscribe message not being sent to all peers causing them to
keep thinking we're still subscribed
* release memory earlier in a few places
* floodsub fix
* add floodPublish test
* test delivery via control Iwant/have mechanics
* fix issues in control, and add testing
* fix possible backoff issue with pruned routine overriding it
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.
In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.
To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)
* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
Tanguy Cizain
markspanbroek
Miran
Dmitriy Ryajov
Ben Tilford
Giovanni Petrantoni
Ștefan Talpalaru
Jacek Sieka