logos-messaging/logos-messaging-nim
1
0
Fork 0
mirror of https://github.com/logos-messaging/logos-messaging-nim.git synced 2026-07-31 20:43:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
logos-messaging-nim/docs/operators/how-to/monitor.md

150 lines
6.6 KiB
Markdown
Raw Normal View History

docs: add monitoring guide for prometheus and grafana (#1168) * docs: add monitoring guide for prometheus and grafana * docs: address comments
2022-09-20 10:09:36 +01:00
# Monitor nwaku using Prometheus and Grafana
## Prerequisites
1. A running nwaku instance with HTTP metrics server enabled (i.e. with `--metrics-server:true`)
2. [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) installed
### Installing Prometheus
Prometheus can be installed by downloading and extracting
the latest release for your system distribution from the [Prometheus download page](https://prometheus.io/download/).
For example, on a DebianOS distribution you could run
```bash
wget https://github.com/prometheus/prometheus/releases/download/v2.38.0/prometheus-2.38.0.linux-amd64.tar.gz
tar xvfz prometheus-2.38.0.linux-amd64.tar.gz
```
For more advanced installations,
Prometheus has a handy [Getting Started](https://prometheus.io/docs/prometheus/latest/getting_started/) page to guide you through the process.
There are also many third party guides on installing Prometheus for specific distributions,
such as [this old but still relevant one](https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04) from DigitalOcean.
We also suggest running Prometheus as a service,
as explained by [this guide](https://www.devopsschool.com/blog/how-to-run-prometheus-server-as-a-service/).
Bear in mind that we'll be creating our own `prometheus.yml` configuration file later on when you encounter this in any of the guides.
### Installing Grafana
Follow the [installation instructions](https://grafana.com/docs/grafana/latest/setup-grafana/installation/) appropriate to your distribution to install Grafana.
The stable version of the Grafana Enterprise Edition is the free, recommended edition to install.
## Configure Prometheus
1. Create a file called `prometheus.yml` with the following content:
```yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'nwaku'
scrape_interval: 1s
static_configs:
- targets: ['localhost:<nwaku_port>']
```
Replace `<nwaku_port>` with the metrics HTTP server port of your running nwaku instance.
For default configurations metrics are reported on port `8008` of the `localhost`.
If you've used `--ports-shift`, or explicitly set the metrics port using `--metrics-server-port`, this port will be different from the default.
It's possible to extract the metrics server port from the startup logs of the nwaku node.
Look for a log with the format below and substitute `nwaku_port` with the value reported after `serverPort=`:
```
INF 2022-09-16 12:14:12.739+01:00 Metrics HTTP server started topics="wakunode.setup.metrics" tid=6243 file=wakunode2_setup_metrics.nim:29 serverIp=127.0.0.1 serverPort=8009
```
2. Start Prometheus using the config file you created above:
```bash
./path/to/prometheus --config.file=/path/to/prometheus.yml &
```
3. Verify that Prometheus is running correctly.
Once Prometheus is running, it exposes by default a management console on port `9090`.
If you are running Prometheus locally, for example,
you can visit http://localhost:9090/ in a browser to view basic info about the running instance.
http://localhost:9090/targets shows the state of the different metrics server endpoints that we configured in `prometheus.yml`.
In our case we'd expect Prometheus to successfully scrape metrics off two endpoints,
the running nwaku instance and Prometheus itself.
## Configure Grafana
1. Start the Grafana server, if it's not running already after installation.
```bash
sudo systemctl start grafana-server
```
2. Open Grafana in your browser.
Grafana exposes its interface by default on port `3000`.
For example, if you are running Grafana locally,
you can find it by navigating to http://localhost:3000/.
If you are prompted for a username and password,
the default is `admin` in both cases.
3. Set Prometheus as your data source.
[These instructions](https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/) describe how to add a new data source.
The default values for setting up a Prometheus data source should be sufficient.
4. Create a new dashboard or import an existing one.
You can now visualize metrics off your running nwaku instance by [creating a new dashboard and adding panels](https://grafana.com/docs/grafana/latest/dashboards/add-organize-panels/) for the metric(s) of your choice.
To get you started,
we have published a [basic monitoring dashboard for a single nwaku node](https://github.com/status-im/nwaku/blob/d4e899fba77389d20ca19c73a9443501039cdef2/metrics/waku-single-node-dashboard.json)
which you can [import to your Grafana instance](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#import-a-dashboard).
5. Happy monitoring!
Some of the most important metrics to keep an eye on include:
- `libp2p_peers` as an indication of how many peers your node is connected to,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074) * refactor(metrics): prefix node metrics with logos_delivery_ Every metric the node exports now starts with logos_delivery_. There was no prefix mechanism before: nim-metrics derives the exported name from the Nim identifier, no declaration passed an explicit `name = "..."`, and the waku_ convention was maintained by hand -- 69 of the 80 node metrics followed it and 11 did not (query_count, query_time_secs, event_loop_load, event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*, total_* and the camelCase rendezvousPeerFoundTotal). Identifiers are renamed rather than given a `name = "..."` argument, keeping the invariant that the Nim identifier is the exported name and letting the compiler check every call site. rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the only camelCase metric, and the trailing Total was redundant since nim-metrics already appends _total to counters at exposition time. library/ is untouched on purpose -- `proc waku_version()` in kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge of the same name in node_telemetry.nim. BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules that reference waku_* must be updated; see docs/operators/how-to/monitor.md. * refactor(metrics): prefix auxiliary app metrics with logos_delivery_ Applies the same prefix to the tools shipped from this repo: liteprotocoltester (lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the lightpush_mix example (lp_mix_*). These tools are not the delivery node and already had their own consistent prefixes, so this commit is separable from the node rename if the intent was to namespace only the node itself. * chore(metrics): query old and new metric names in Grafana dashboards 212 expressions across 9 dashboards now match both the waku_* and the logos_delivery_* spelling, so panels keep working across the upgrade and over historical data: sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval]) or increase(logos_delivery_node_errors_total{...}[$__rate_interval]))) The `or` is placed around the leaf, inside every aggregation. That depth is load-bearing: `or` keeps its right operand only for label sets absent from the left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the fleet separately and then discards the right one entirely -- silently dropping every already-upgraded node. 36 panels here collapse `instance`. Measured against a local Prometheus scraping two targets, one exporting old names at 10/s and one exporting new names at 20/s (truth 30/s): union outside the aggregation gives 10, union around the leaf gives 30. Where the leaf sits in a range vector the whole call is duplicated, since `(a or b)[5m]` is not valid PromQL. Once every scraped node runs a release with the new names and the old samples have aged out of retention, the `or` half can be deleted. * test(e2e): expect logos_delivery_-prefixed metric names The e2e suite asserts against a live /metrics endpoint, which serves only the new names, so these are replaced rather than unioned. libp2p_* entries are unchanged. * docs(operators): document the logos_delivery_ metric prefix Records that every metric the node exports is prefixed, that dependency metrics (libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or` has to sit if operators maintain their own dashboards or alert rules. * refactor(metrics): name the store fleet metrics after store, not relay logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in waku_store/protocol_metrics.nim and recorded by the store client, but carried a relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and logos_delivery_store_fleet_msg_count. The dashboard keeps matching the old exported name, which was waku_relay_fleet_store_*. Note that both metrics are wrong independently of their name, see the PR description.
2026-07-29 18:56:50 +01:00
- `logos_delivery_node_messages_total` to view the total amount of network traffic relayed by your node and
- `logos_delivery_node_errors` as a rough indication of basic operating errors logged by the node.
## Metric names
Every metric exported by the node is prefixed with `logos_delivery_`.
Metrics coming from libraries the node builds on (`libp2p_*`, `nim_gc_*`, `process_*`)
keep their own names and are unaffected.
Nodes before this rename exported the same metrics under a `waku_` prefix (and a few
under no prefix at all, such as `query_count` or `event_loop_load`). The dashboards in
[`metrics/`](https://github.com/logos-messaging/logos-delivery/tree/master/metrics) query
both spellings, so they keep working across the upgrade and over historical data:
```
(waku_store_queries_total or logos_delivery_store_queries_total)
```
Place the `or` around the leaf, inside any aggregation — not around the whole
expression. `or` keeps its right operand only for label sets that are absent from the
left, so this undercounts while a fleet is mid-upgrade:
```
sum by (type)(waku_node_errors_total) or sum by (type)(logos_delivery_node_errors_total)
```
Both sides collapse to the same label set, so the right one is discarded entirely and
every already-upgraded node vanishes from the panel. Write this instead:
```
sum by (type)(waku_node_errors_total or logos_delivery_node_errors_total)
```
When the metric sits inside a range vector, duplicate the whole call —
`(rate(old[5m]) or rate(new[5m]))` — since `(old or new)[5m]` is not valid PromQL.
If you maintain your own dashboards, alert rules or recording rules, apply the same
change before upgrading. Once every node you scrape runs a release with the new names
and the old samples have aged out of your Prometheus retention window, the `waku_`
alternative can be dropped.
docs: add monitoring guide for prometheus and grafana (#1168) * docs: add monitoring guide for prometheus and grafana * docs: address comments
2022-09-20 10:09:36 +01:00
Reference in New Issue Copy Permalink