mirror of https://github.com/logos-messaging/logos-delivery.git synced 2026-07-30 00:13:32 +00:00

refactor(metrics): give every metric a logos_delivery_ prefix (#4074 )

* refactor(metrics): prefix node metrics with logos_delivery_

Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).

Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.

rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.

library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.

BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.

* refactor(metrics): prefix auxiliary app metrics with logos_delivery_

Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).

These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.

* chore(metrics): query old and new metric names in Grafana dashboards

212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:

  sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
              or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))

The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.

Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.

Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.

Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.

* test(e2e): expect logos_delivery_-prefixed metric names

The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.

* docs(operators): document the logos_delivery_ metric prefix

Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.

* refactor(metrics): name the store fleet metrics after store, not relay

logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.

The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.

Note that both metrics are wrong independently of their name, see the PR
description.

2026-07-29 18:56:50 +01:00

6.6 KiB

Raw Blame History

Monitor nwaku using Prometheus and Grafana

Prerequisites

A running nwaku instance with HTTP metrics server enabled (i.e. with --metrics-server:true)
Prometheus and Grafana installed

Installing Prometheus

Prometheus can be installed by downloading and extracting the latest release for your system distribution from the Prometheus download page.

For example, on a DebianOS distribution you could run

wget https://github.com/prometheus/prometheus/releases/download/v2.38.0/prometheus-2.38.0.linux-amd64.tar.gz
tar xvfz prometheus-2.38.0.linux-amd64.tar.gz

For more advanced installations, Prometheus has a handy Getting Started page to guide you through the process. There are also many third party guides on installing Prometheus for specific distributions, such as this old but still relevant one from DigitalOcean. We also suggest running Prometheus as a service, as explained by this guide. Bear in mind that we'll be creating our own prometheus.yml configuration file later on when you encounter this in any of the guides.

Installing Grafana

Follow the installation instructions appropriate to your distribution to install Grafana. The stable version of the Grafana Enterprise Edition is the free, recommended edition to install.

Configure Prometheus

Create a file called prometheus.yml with the following content:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'nwaku'
    scrape_interval: 1s
    static_configs:
      - targets: ['localhost:<nwaku_port>']

Replace <nwaku_port> with the metrics HTTP server port of your running nwaku instance. For default configurations metrics are reported on port 8008 of the localhost. If you've used --ports-shift, or explicitly set the metrics port using --metrics-server-port, this port will be different from the default. It's possible to extract the metrics server port from the startup logs of the nwaku node. Look for a log with the format below and substitute nwaku_port with the value reported after serverPort=:

INF 2022-09-16 12:14:12.739+01:00 Metrics HTTP server started                topics="wakunode.setup.metrics" tid=6243 file=wakunode2_setup_metrics.nim:29 serverIp=127.0.0.1 serverPort=8009

Start Prometheus using the config file you created above:

./path/to/prometheus --config.file=/path/to/prometheus.yml &

Verify that Prometheus is running correctly.

Once Prometheus is running, it exposes by default a management console on port 9090. If you are running Prometheus locally, for example, you can visit http://localhost:9090/ in a browser to view basic info about the running instance. http://localhost:9090/targets shows the state of the different metrics server endpoints that we configured in prometheus.yml. In our case we'd expect Prometheus to successfully scrape metrics off two endpoints, the running nwaku instance and Prometheus itself.

Configure Grafana

Start the Grafana server, if it's not running already after installation.

sudo systemctl start grafana-server

Open Grafana in your browser.

Grafana exposes its interface by default on port 3000. For example, if you are running Grafana locally, you can find it by navigating to http://localhost:3000/. If you are prompted for a username and password, the default is admin in both cases.

Set Prometheus as your data source.

These instructions describe how to add a new data source. The default values for setting up a Prometheus data source should be sufficient.

Create a new dashboard or import an existing one.

You can now visualize metrics off your running nwaku instance by creating a new dashboard and adding panels for the metric(s) of your choice. To get you started, we have published a basic monitoring dashboard for a single nwaku node which you can import to your Grafana instance.

Happy monitoring!

Some of the most important metrics to keep an eye on include:

libp2p_peers as an indication of how many peers your node is connected to,
logos_delivery_node_messages_total to view the total amount of network traffic relayed by your node and
logos_delivery_node_errors as a rough indication of basic operating errors logged by the node.

Metric names

Every metric exported by the node is prefixed with logos_delivery_. Metrics coming from libraries the node builds on (libp2p_*, nim_gc_*, process_*) keep their own names and are unaffected.

Nodes before this rename exported the same metrics under a waku_ prefix (and a few under no prefix at all, such as query_count or event_loop_load). The dashboards in metrics/ query both spellings, so they keep working across the upgrade and over historical data:

(waku_store_queries_total or logos_delivery_store_queries_total)

Place the or around the leaf, inside any aggregation — not around the whole expression. or keeps its right operand only for label sets that are absent from the left, so this undercounts while a fleet is mid-upgrade:

sum by (type)(waku_node_errors_total) or sum by (type)(logos_delivery_node_errors_total)

Both sides collapse to the same label set, so the right one is discarded entirely and every already-upgraded node vanishes from the panel. Write this instead:

sum by (type)(waku_node_errors_total or logos_delivery_node_errors_total)

When the metric sits inside a range vector, duplicate the whole call — (rate(old[5m]) or rate(new[5m])) — since (old or new)[5m] is not valid PromQL.

If you maintain your own dashboards, alert rules or recording rules, apply the same change before upgrading. Once every node you scrape runs a release with the new names and the old samples have aged out of your Prometheus retention window, the waku_ alternative can be dropped.

6.6 KiB Raw Blame History