nim-libp2p-experimental/libp2p
Jacek Sieka e285d8bbf4
mem usage cleanups for pubsub (#564)
In `async` functions, a closure environment is created for variables
that cross an await boundary - this closure environment is kept in
memory for the lifetime of the associated future - this means that
although _some_ variables are no longer used, they still take up memory
for a long time.

In Nimbus, message validation is processed in batches meaning the future
of an incoming gossip message stays around for quite a while - this
leads to memory consumption peaks of 100-200 mb when there are many
attestations in the pipeline.

To avoid excessive memory usage, it's generally better to move non-async
code into proc's such that the variables therein can be released earlier
- this includes the many hidden variables introduced by macro and
template expansion (ie chronicles that does expensive exception
handling)

* move seen table salt to floodsub, use there as well
* shorten seen table salt to size of hash
* avoid unnecessary memory allocations and copies in a few places
* factor out message scoring
* avoid reencoding outgoing message for every peer
* keep checking validators until reject (in case there's both reject and
ignore)
* `readOnce` avoids `readExactly` overhead for single-byte read
* genericAssign -> assign2
2021-04-18 10:08:33 +02:00
..
crypto Debt: cleanup warnings #426 (#536) 2021-02-25 09:24:49 -06:00
daemon fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
muxers fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
protobuf Add `libp2p_dump` and `libp2p_dump_dir` compiler time options. (#359) 2020-09-15 11:16:43 +02:00
protocols mem usage cleanups for pubsub (#564) 2021-04-18 10:08:33 +02:00
stream mem usage cleanups for pubsub (#564) 2021-04-18 10:08:33 +02:00
transports Fix minor exception issues (#550) 2021-03-23 07:45:25 +01:00
upgrademngrs Connection limits (#384) 2021-01-20 22:00:24 -06:00
utils Semaphore cancellations (#503) 2021-01-14 10:11:12 +01:00
builders.nim More builders (#560) 2021-04-06 14:16:23 -06:00
cid.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
connmanager.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
debugutils.nim fix debugutils (#423) 2020-11-04 19:56:28 -06:00
errors.nim warn -> debug log levels in errors.nim 2020-08-20 16:53:28 +09:00
multiaddress.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multibase.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multicodec.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multihash.nim fix peer score accumulation (#541) 2021-03-09 13:22:52 +01:00
multistream.nim adding libp2p tag to logScope (#465) 2020-12-01 11:34:27 -06:00
peerid.nim clean up peerid (#470) 2020-12-03 13:53:16 -06:00
peerinfo.nim simplify and unify logging (#353) 2020-09-06 10:31:47 +02:00
peerstore.nim Merge unstable (#518) 2021-02-08 15:16:23 -06:00
standard_switch.nim More builders (#560) 2021-04-06 14:16:23 -06:00
switch.nim Fix minor exception issues (#550) 2021-03-23 07:45:25 +01:00
transcoder.nim add side effect annotations (#197) 2020-06-01 09:25:16 +02:00
utility.nim now wild `except:` 2021-03-15 16:48:11 +09:00
varint.nim use stew/leb128 (#481) 2020-12-15 12:15:22 -06:00
vbuffer.nim use stew/leb128 (#481) 2020-12-15 12:15:22 -06:00
wire.nim Rework transport to use chronos accept (#420) 2020-11-18 20:06:42 -06:00