e285d8bbf4
In `async` functions, a closure environment is created for variables that cross an await boundary - this closure environment is kept in memory for the lifetime of the associated future - this means that although _some_ variables are no longer used, they still take up memory for a long time. In Nimbus, message validation is processed in batches meaning the future of an incoming gossip message stays around for quite a while - this leads to memory consumption peaks of 100-200 mb when there are many attestations in the pipeline. To avoid excessive memory usage, it's generally better to move non-async code into proc's such that the variables therein can be released earlier - this includes the many hidden variables introduced by macro and template expansion (ie chronicles that does expensive exception handling) * move seen table salt to floodsub, use there as well * shorten seen table salt to size of hash * avoid unnecessary memory allocations and copies in a few places * factor out message scoring * avoid reencoding outgoing message for every peer * keep checking validators until reject (in case there's both reject and ignore) * `readOnce` avoids `readExactly` overhead for single-byte read * genericAssign -> assign2 |
||
|---|---|---|
| .github/workflows | ||
| docs | ||
| examples | ||
| libp2p | ||
| scripts | ||
| tests | ||
| tools | ||
| .appveyor.yml | ||
| .gitignore | ||
| LICENSE | ||
| LICENSE-APACHEv2 | ||
| LICENSE-MIT | ||
| README.md | ||
| codecov.yml | ||
| libp2p.nim | ||
| libp2p.nimble | ||
README.md
The Nim implementation of the libp2p Networking Stack.
Introduction
An implementation of libp2p in Nim. Also provides a Nim wrapper of the Libp2p Go daemon.
Project Status
The current native Nim libp2p implementation support is experimental and shouldn't be relied on for production use. It is under active development and contributions are highly welcomed. :)
Check our examples folder to get started!
Table of Contents
Background
libp2p is a networking stack and library modularized out of The IPFS Project, and bundled separately for other tools to use.
libp2p is the product of a long and arduous quest of understanding; a deep dive into the internet's network stack and the peer-to-peer protocols from the past. Building large scale peer-to-peer systems has been complex and difficult in the last 15 years and libp2p is a way to fix that. It is a "network stack", a suite of networking protocols that cleanly separates concerns and enables sophisticated applications to only use the protocols they absolutely need, without giving up interoperability and upgradeability.
libp2p grew out of IPFS, but it is built so that lots of people can use it, for lots of different projects.
- Learn more about libp2p at libp2p.io and follow our evolving documentation efforts at docs.libp2p.io.
- Here is an overview of libp2p and its implementations in other programming languages.
Install
nimble install libp2p
Prerequisite
Usage
API
The specification is available in the docs/api folder.
Getting Started
Please read the GETTING_STARTED.md guide.
Tutorials and Examples
Example code can be found in the examples folder.
Direct Chat Tutorial
- Part I: Set up the main function and use multi-thread for processing IO.
- Part II: Dial remote peer and allow customized user input commands.
- Part III: Configure and establish a libp2p node.
Using the Go Daemon
Please find the installation and usage intructions in daemonapi.md.
Examples can be found in the examples/go-daemon folder;
Development
Clone and Install dependencies:
git clone https://github.com/status-im/nim-libp2p
cd nim-libp2p
nimble install
Tests
Prerequisite
Run unit tests
# run all the unit tests
nimble test
Packages
List of packages currently in existence for nim-libp2p:
Libp2p
Transports
Secure Channels
Stream Multiplexers
Utilities
Data Types
Pubsub
Packages that exist in the original libp2p specs and are under active development:
- libp2p-daemon
- libp2p-webrtc-direct
- libp2p-webrtc-star
- libp2p-websockets
- libp2p-spdy
- libp2p-bootstrap
- libp2p-kad-dht
- libp2p-mdns
- libp2p-webrtc-star
- libp2p-delegated-content-routing
- libp2p-delegated-peer-routing
- libp2p-nat-mgnr
- libp2p-utils
** Note that the current stack reflects the minimal requirements for the upcoming Eth2 implementation.
Tips and tricks
enable expensive metrics:
nim c -d:libp2p_expensive_metrics some_file.nim
use identify metrics
nim c -d:libp2p_agents_metrics -d:KnownLibP2PAgents=nimbus,lighthouse,prysm,teku some_file.nim
specify gossipsub specific topics to measure
nim c -d:KnownLibP2PTopics=topic1,topic2,topic3 some_file.nim
Contribute
The libp2p implementation in Nim is a work in progress. We welcome contributors to help out! Specifically, you can:
- Go through the modules and check out existing issues. This would be especially useful for modules in active development. Some knowledge of IPFS/libp2p may be required, as well as the infrastructure behind it.
- Perform code reviews. Feel free to let us know if you found anything that can a) speed up the project development b) ensure better quality and c) reduce possible future bugs.
- Add tests. Help nim-libp2p to be more robust by adding more tests to the tests folder.
Core Developers
@cheatfate, Dmitriy Ryajov, Giovanni Petrantoni, Zahary Karadjov
License
Licensed and distributed under either of
- MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT
or
- Apache License, Version 2.0, (LICENSE-APACHEv2 or http://www.apache.org/licenses/LICENSE-2.0)
at your option. These files may not be copied, modified, or distributed except according to those terms.