nim-libp2p/libp2p
Dmitriy Ryajov c949f14a99
Merge master to unstable (#570)
* Revisit Floodsub (#543)

Fixes #525

add coverage to unsubscribeAll and testing

* add mounted protos to identify message (#546)

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump (#549)

* Fix minor exception issues (#550)

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* use a builder pattern to build the switch (#551)

* use a builder pattern to build the switch

* with with

* more refs

* builders (#559)

* More builders (#560)

* address some issues pointed out in review

* re-add to prevent breaking other projects

* mem usage cleanups for pubsub (#564)

In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.

In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.

To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)

* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2

* More gossip coverage (#553)

* add floodPublish test

* test delivery via control Iwant/have mechanics

* fix issues in control, and add testing

* fix possible backoff issue with pruned routine overriding it

* fix control messages (#566)

* remove unused control graft check in handleControl

* avoid sending empty Iwant messages

* Split dialer (#542)

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* Merge master (#555)

* Revisit Floodsub (#543)

Fixes #525

add coverage to unsubscribeAll and testing

* add mounted protos to identify message (#546)

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump (#549)

* Fix minor exception issues (#550)

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer (#542)

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* Builders (#558)

* use a builder pattern to build the switch (#551)

* use a builder pattern to build the switch

* with with

* more refs

* Merge master (#555)

* Revisit Floodsub (#543)

Fixes #525

add coverage to unsubscribeAll and testing

* add mounted protos to identify message (#546)

* add stable/unstable auto bumps

* fix auto-bump CI

* merge nbc auto bump with CI in order to bump only on CI success

* put conditional locks on nbc bump (#549)

* Fix minor exception issues (#550)

Makes code compatible with
https://github.com/status-im/nim-chronos/pull/166 without requiring it.

* fix nimbus ref for auto-bump stable's PR

* Split dialer (#542)

* extracting dialing logic to dialer

* exposing upgrade methods on transport

* cleanup

* fixing tests to use new interfaces

* add comments

* add base exception class and fix hierarchy

* fix imports

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

* `doAssert` is `ValueError` not `AssertionError`?

* revert back to `AssertionError`

* fix builders

* more builder stuff

* more builders

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>

Co-authored-by: Giovanni Petrantoni <7008900+sinkingsugar@users.noreply.github.com>
Co-authored-by: Jacek Sieka <jacek@status.im>
2021-06-02 12:24:46 -06:00
..
crypto Debt: cleanup warnings #426 (#536) 2021-02-25 09:24:49 -06:00
daemon add base exception class and fix hierarchy 2021-06-02 12:24:04 -06:00
muxers fix imports 2021-06-02 12:24:06 -06:00
protobuf Add `libp2p_dump` and `libp2p_dump_dir` compiler time options. (#359) 2020-09-15 11:16:43 +02:00
protocols Merge master to unstable (#570) 2021-06-02 12:24:46 -06:00
stream fix imports 2021-06-02 12:24:06 -06:00
transports Split dialer (#542) 2021-06-02 12:23:44 -06:00
upgrademngrs Split dialer (#542) 2021-06-02 12:23:44 -06:00
utils adding raises defect across the codebase (#572) 2021-05-21 10:27:01 -06:00
builders.nim Merge master (#562) 2021-06-02 12:24:46 -06:00
cid.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
connmanager.nim use LPError more consistently (#582) 2021-06-02 15:39:10 +02:00
debugutils.nim fix debugutils (#423) 2020-11-04 19:56:28 -06:00
dial.nim Split dialer (#542) 2021-06-02 12:23:44 -06:00
dialer.nim Split dialer (#542) 2021-06-02 12:23:44 -06:00
errors.nim use LPError more consistently (#582) 2021-06-02 15:39:10 +02:00
multiaddress.nim use LPError more consistently (#582) 2021-06-02 15:39:10 +02:00
multibase.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multicodec.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multihash.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multistream.nim use LPError more consistently (#582) 2021-06-02 15:39:10 +02:00
peerid.nim clean up peerid (#470) 2020-12-03 13:53:16 -06:00
peerinfo.nim use LPError more consistently (#582) 2021-06-02 15:39:10 +02:00
peerstore.nim Merge unstable (#518) 2021-02-08 15:16:23 -06:00
standard_switch.nim More builders (#560) 2021-04-06 14:16:23 -06:00
switch.nim Split dialer (#542) 2021-06-02 12:23:44 -06:00
transcoder.nim add side effect annotations (#197) 2020-06-01 09:25:16 +02:00
utility.nim now wild `except:` 2021-03-15 16:48:11 +09:00
varint.nim use stew/leb128 (#481) 2020-12-15 12:15:22 -06:00
vbuffer.nim use stew/leb128 (#481) 2020-12-15 12:15:22 -06:00
wire.nim use LPError more consistently (#582) 2021-06-02 15:39:10 +02:00