mirror of
https://github.com/logos-messaging/logos-messaging-nim.git
synced 2026-07-30 12:03:30 +00:00
8125cd0262
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
networkmonitor
Monitoring tool to run in an existing waku network with the following features:
- Keeps discovering new peers using
discv5 - Tracks advertised capabilities of each node as per stored in the ENR
wakufield - Attempts to connect to all nodes, tracking which protocols each node supports
- Presents grafana-ready metrics showing the state of the network in terms of locations, ips, number discovered peers, number of peers we could connect to, user-agent that each peer contains, content topics and the amount of rx messages in each one.
- Metrics are exposed through prometheus metrics but also with a custom rest api, presenting detailed information about each peer. These metrics are exposed via a rest api.
Usage
./build/networkmonitor --help
Usage:
networkmonitor [OPTIONS]...
The following options are available:
-l, --log-level Sets the log level [=LogLevel.INFO].
-t, --timeout Timeout to consider that the connection failed [=chronos.seconds(10)].
-b, --bootstrap-node Bootstrap ENR node. Argument may be repeated. [=@[""]].
--dns-discovery-url URL for DNS node list in format 'enrtree://<key>@<fqdn>'.
--pubsub-topic Default pubsub topic to subscribe to. Argument may be repeated..
-r, --refresh-interval How often new peers are discovered and connected to (in seconds) [=5].
--cluster-id Cluster id that the node is running in. Node in a different cluster id is
disconnected. [=1].
--rln-relay Enable spam protection through rln-relay: true|false [=true].
--rln-relay-dynamic Enable waku-rln-relay with on-chain dynamic group management: true|false
[=true].
--rln-relay-eth-client-address HTTP address of an Ethereum testnet client e.g., http://localhost:8540/
[=http://localhost:8540/].
--rln-relay-eth-contract-address Address of membership contract on an Ethereum testnet.
--rln-relay-epoch-sec Epoch size in seconds used to rate limit RLN memberships. Default is 1 second.
[=1].
--rln-relay-user-message-limit Set a user message limit for the rln membership registration. Must be a positive
integer. Default is 1. [=1].
--metrics-server Enable the metrics server: true|false [=true].
--metrics-server-address Listening address of the metrics server. [=parseIpAddress("127.0.0.1")].
--metrics-server-port Listening HTTP port of the metrics server. [=8008].
--metrics-rest-address Listening address of the metrics rest server. [=127.0.0.1].
--metrics-rest-port Listening HTTP port of the metrics rest server. [=8009].
Example
Connect to the network through a given bootstrap node, with default parameters. See metrics section for the data that it exposes.
./build/networkmonitor --log-level=INFO --b="enr:-QEkuEB3WHNS-xA3RDpfu9A2Qycr3bN3u7VoArMEiDIFZJ66F1EB3d4wxZN1hcdcOX-RfuXB-MQauhJGQbpz3qUofOtLAYJpZIJ2NIJpcIQI2SVcim11bHRpYWRkcnO4bgA0Ni9ub2RlLTAxLmFjLWNuLWhvbmdrb25nLWMud2FrdS5zYW5kYm94LnN0YXR1cy5pbQZ2XwA2Ni9ub2RlLTAxLmFjLWNuLWhvbmdrb25nLWMud2FrdS5zYW5kYm94LnN0YXR1cy5pbQYfQN4DgnJzkwABCAAAAAEAAgADAAQABQAGAAeJc2VjcDI1NmsxoQPK35Nnz0cWUtSAhBp7zvHEhyU_AqeQUlqzLiLxfP2L4oN0Y3CCdl-DdWRwgiMohXdha3UyDw"
./build/networkmonitor --log-level=INFO --dns-discovery-url=enrtree://AL65EKLJAUXKKPG43HVTML5EFFWEZ7L4LOKTLZCLJASG4DSESQZEC@prod.status.nodes.status.im
Metrics
Metrics are divided into two categories:
- Prometheus metrics, exposed as i.e. gauges.
- Custom metrics, used for unconstrained labels such as peer information or content topics.
- These metrics are not exposed through prometheus because since they are unconstrained, they can end up breaking the backend, as a new datapoint is generated for each one and it can reach up a point where is too much to handle.
### Prometheus Metrics
The following metrics are available. See http://localhost:8008/metrics
peer_type_as_per_enr: Number of peers supporting each capability according to the ENR (Relay, Store, Lightpush, Filter)peer_type_as_per_protocol: Number of peers supporting each protocol, after a successful connection)peer_user_agents: List of useragents found in the network and their count
Other relevant metrics reused from nim-eth:
routing_table_nodes: Inherited from nim-eth, number of nodes in the routing tablediscovery_message_requests_outgoing_total: Inherited from nim-eth, number of outgoing discovery requests, useful to know if the node is actively looking for new peers
### Custom Metrics
The following endpoints are available:
http://localhost:8009/allpeersinfo: json list of all peers with extra information such as ip, location, supported protocols and last connection time.http://localhost:8009/contenttopics: content topic messages and its message count.