nwaku/waku
Abhimanyu 8897ae1a2f
chore: add size retention policy (#2093)
* chore: add retention policy with GB or MB limitation #1885

* chore: add retention policy with GB or MB limitation

* chore: updated code post review- retention policy

* ci: extract discordNotify to separate file

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: push images to new wakuorg/nwaku repo

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: enforce default Docker image tags strictly

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: push GIT_REF if it looks like a version

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* fix: update wakuv2 fleet DNS discovery enrtree

https://github.com/status-im/infra-misc/issues/171

* chore: resolving DNS IP and publishing it when no extIp is provided (#2030)

* feat(coverage): Add simple coverage (#2067)

* Add test aggregator to all directories.
* Implement coverage script.

* fix(ci): fix name of discord notify method

Also use absolute path to load Groovy script.

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version (#2080)

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version

* add more metrics, refactor how most metrics are calculated

* rework metrics table fillup

* reset connErr to make sure we honour successful reconnection

* chore(cbindings): Adding cpp example that integrates the 'libwaku' (#2079)

* Adding cpp example that integrates the `libwaku`

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>

* fix(ci): update the dependency list in pre-release WF (#2088)

* chore: adding NetConfig test suite (#2091)

---------

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Anton Iakimov <yakimant@gmail.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>
Co-authored-by: Vaclav Pavlin <vaclav@status.im>
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2023-09-30 11:10:52 +05:30
..
common chore: add size retention policy (#2093) 2023-09-30 11:10:52 +05:30
node feat: Autosharding API for RELAY subscriptions (#1983) 2023-09-26 07:33:52 -04:00
utils chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_api feat: added RELAY openapi definitions (#2081) 2023-09-26 15:53:46 -04:00
waku_archive chore: add size retention policy (#2093) 2023-09-30 11:10:52 +05:30
waku_core feat: Autosharding API for RELAY subscriptions (#1983) 2023-09-26 07:33:52 -04:00
waku_enr feat: topic subscriptions updates discv5 ENR (#1875) 2023-08-23 09:53:17 -04:00
waku_filter feat: HTTP REST API: Filter support v2 (#1890) 2023-09-14 21:28:57 +02:00
waku_filter_v2 chore: Updating nim-chronicles, nim-chronos, nim-presto, nimcrypto, nim-libp2p, and nim-nat-transversal (#2043) 2023-09-18 18:51:49 +02:00
waku_keystore chore(rln-relay): use the only key from keystore if only 1 exists (#1984) 2023-09-04 15:46:44 +05:30
waku_lightpush feat: update various protocols to autoshard (#1857) 2023-08-17 08:11:18 -04:00
waku_noise chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_peer_exchange chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_relay feat: Autosharding API for RELAY subscriptions (#1983) 2023-09-26 07:33:52 -04:00
waku_rln_relay fix(rln-relay): segfault when no params except rln-relay is passed in (#2047) 2023-09-25 13:26:16 +05:30
waku_store chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
README.md fix: update wakuv2 fleet DNS discovery enrtree 2023-09-27 13:39:10 +02:00
waku.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_api.nim refactor: Moved external APIs out of node (#2069) 2023-09-22 09:36:46 -04:00
waku_archive.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_core.nim chore: move SubscriptionManager under waku_core (#2025) 2023-09-12 17:08:40 +02:00
waku_discv5.nim feat: Autosharding API for RELAY subscriptions (#1983) 2023-09-26 07:33:52 -04:00
waku_dnsdisc.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_enr.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_filter.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_filter_v2.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_keystore.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_lightpush.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_node.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_peer_exchange.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_relay.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_rln_relay.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00
waku_store.nim chore: remove references to v2 (#1898) 2023-08-09 18:11:50 +01:00

README.md

Waku

This folder contains code related to Waku, both as a node and as a protocol.

Introduction

This is an implementation in Nim of the Waku suite of protocols.

See specifications.

How to Build & Run

Prerequisites

  • GNU Make, Bash and the usual POSIX utilities. Git 2.9.4 or newer.

Wakunode binary

# The first `make` invocation will update all Git submodules.
# You'll run `make update` after each `git pull`, in the future, to keep those submodules up to date.
make wakunode2

# See available command line options
./build/wakunode2 --help

# Connect the client directly with the Status test fleet
# TODO NYI
#./build/wakunode2 --log-level:debug --discovery:off --fleet:test --log-metrics

Note: building wakunode2 requires 2GB of RAM. The build will fail on systems not fulfilling this requirement.

Setting up a wakunode2 on the smallest digital ocean droplet, you can either

  • compile on a stronger droplet featuring the same CPU architecture and downgrade after compiling, or
  • activate swap on the smallest droplet, or
  • use Docker.

Waku Protocol Test Suite

# Run all the Waku tests
make test

To run a specific test.

# Get a shell with the right environment variables set
./env.sh bash
# Run a specific test
nim c -r ./tests/test_waku_filter_legacy.nim

You can also alter compile options. For example, if you want a less verbose output you can do the following. For more, refer to the compiler flags and chronicles documentation.

nim c -r -d:chronicles_log_level=WARN --verbosity=0 --hints=off ./tests/waku_filter_v2/test_waku_filter.nim

You may also want to change the outdir to a folder ignored by git.

nim c -r -d:chronicles_log_level=WARN --verbosity=0 --hints=off --outdir=build ./tests/waku_filter_v2/test_waku_filter.nim

Waku Protocol Example

There are basic examples of both publishing and subscribing, more limited in features and configuration than the wakunode2 binary, located in examples/.

There is also a more full featured example in apps/chat2/.

Using Metrics

Metrics are available for Waku nodes.

make wakunode2
./build/wakunode2 --metrics-server

Ensure your Prometheus config prometheus.yml contains the targets you care about, e.g.:

scrape_configs:
  - job_name: "waku"
    static_configs:
      - targets: ['localhost:8008', 'localhost:8009', 'localhost:8010']

For visualisation, similar steps can be used as is written down for Nimbus here.

There is a similar example dashboard that includes visualisation of the envelopes available at metrics/waku-grafana-dashboard.json.

Spec support

All Waku RFCs reside at rfc.vac.dev. Note that Waku specs are titled WAKU2-XXX to differentiate them from a previous legacy version of Waku with RFC titles in the format WAKU-XXX. The legacy Waku protocols are stable, but not under active development.

Generating and configuring a private key

By default a node will generate a new, random key pair each time it boots, resulting in a different public libp2p multiaddrs after each restart.

To maintain consistent addressing across restarts, it is possible to configure the node with a previously generated private key using the --nodekey option.

wakunode2 --nodekey=<64_char_hex>

This option takes a Secp256k1 private key in 64 char hexstring format.

To generate such a key on Linux systems, use the openssl rand command to generate a pseudo-random 32 byte hexstring.

openssl rand -hex 32

Example output:

$ openssl rand -hex 32
6a29e767c96a2a380bb66b9a6ffcd6eb54049e14d796a1d866307b8beb7aee58

where the key 6a29e767c96a2a380bb66b9a6ffcd6eb54049e14d796a1d866307b8beb7aee58 can be used as nodekey.

To create a reusable keyfile on Linux using openssl, use the ecparam command coupled with some standard utilities whenever you want to extract the 32 byte private key in hex format.

# Generate keyfile
openssl ecparam -genkey -name secp256k1 -out my_private_key.pem
# Extract 32 byte private key
openssl ec -in my_private_key.pem -outform DER | tail -c +8 | head -c 32| xxd -p -c 32

Example output:

read EC key
writing EC key
0c687bb8a7984c770b566eae08520c67f53d302f24b8d4e5e47cc479a1e1ce23

where the key 0c687bb8a7984c770b566eae08520c67f53d302f24b8d4e5e47cc479a1e1ce23 can be used as nodekey.

wakunode2 --nodekey=0c687bb8a7984c770b566eae08520c67f53d302f24b8d4e5e47cc479a1e1ce23

Configuring a domain name

It is possible to configure an IPv4 DNS domain name that resolves to the node's public IPv4 address.

wakunode2 --dns4-domain-name=mynode.example.com

This allows for the node's publically announced multiaddrs to use the /dns4 scheme. In addition, nodes with domain name and secure websocket configured, will generate a discoverable ENR containing the /wss multiaddr with /dns4 domain name. This is necessary to verify domain certificates when connecting to this node over secure websocket.

Using DNS discovery to connect to existing nodes

A node can discover other nodes to connect to using DNS-based discovery. The following command line options are available:

--dns-discovery              Enable DNS Discovery
--dns-discovery-url          URL for DNS node list in format 'enrtree://<key>@<fqdn>'
--dns-discovery-name-server  DNS name server IPs to query. Argument may be repeated.
  • --dns-discovery is used to enable DNS discovery on the node. Waku DNS discovery is disabled by default.
  • --dns-discovery-url is mandatory if DNS discovery is enabled. It contains the URL for the node list. The URL must be in the format enrtree://<key>@<fqdn> where <fqdn> is the fully qualified domain name and <key> is the base32 encoding of the compressed 32-byte public key that signed the list at that location.
  • --dns-discovery-name-server is optional and contains the IP(s) of the DNS name servers to query. If left unspecified, the Cloudflare servers 1.1.1.1 and 1.0.0.1 will be used by default.

A node will attempt connection to all discovered nodes.

This can be used, for example, to connect to one of the existing fleets. Current URLs for the published fleet lists:

  • production fleet: enrtree://ANEDLO25QVUGJOUTQFRYKWX6P4Z4GKVESBMHML7DZ6YK4LGS5FC5O@prod.wakuv2.nodes.status.im
  • test fleet: enrtree://AO47IDOLBKH72HIZZOXQP6NMRESAN7CHYWIBNXDXWRJRZWLODKII6@test.wakuv2.nodes.status.im

See the separate tutorial for a complete guide to DNS discovery.

Enabling Websocket

Websocket is currently the only Waku transport supported by browser nodes that uses js-waku. Setting up websocket enables your node to directly serve browser peers.

A valid certificate is necessary to serve browser nodes, you can use letsencrypt:

sudo letsencrypt -d <your.domain.name>

You will need the privkey.pem and fullchain.pem files.

To enable secure websocket, pass the generated files to wakunode2: Note, the default port for websocket is 8000.

wakunode2 --websocket-secure-support=true --websocket-secure-key-path="<letsencrypt cert dir>/privkey.pem" --websocket-secure-cert-path="<letsencrypt cert dir>/fullchain.pem"

Self-signed certificates

Self-signed certificates are not recommended for production setups because:

  • Browsers do not accept self-signed certificates
  • Browsers do not display an error when rejecting a certificate for websocket.

However, they can be used for local testing purposes:

mkdir -p ./ssl_dir/
openssl req -x509 -newkey rsa:4096 -keyout ./ssl_dir/key.pem -out ./ssl_dir/cert.pem -sha256 -nodes
wakunode2 --websocket-secure-support=true --websocket-secure-key-path="./ssl_dir/key.pem" --websocket-secure-cert-path="./ssl_dir/cert.pem"