2021-08-19 09:57:53 +02:00
{
"annotations" : {
"list" : [
{
"builtIn" : 1 ,
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "datasource" ,
"uid" : "grafana"
} ,
2021-08-19 09:57:53 +02:00
"enable" : true ,
"hide" : true ,
"iconColor" : "rgba(0, 211, 255, 1)" ,
"name" : "Annotations & Alerts" ,
2022-02-08 12:22:06 +01:00
"target" : {
"limit" : 100 ,
"matchAny" : false ,
"tags" : [ ] ,
"type" : "dashboard"
} ,
2021-08-19 09:57:53 +02:00
"type" : "dashboard"
}
]
} ,
2025-08-01 16:11:32 +05:30
"description" : "Metrics for Waku nodes written in Nim" ,
2021-08-19 09:57:53 +02:00
"editable" : true ,
2022-02-08 12:22:06 +01:00
"fiscalYearStartMonth" : 0 ,
2025-08-01 16:11:32 +05:30
"graphTooltip" : 1 ,
"id" : 58 ,
2021-08-19 09:57:53 +02:00
"links" : [ ] ,
"panels" : [
{
"collapsed" : false ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 0
} ,
"id" : 56 ,
"panels" : [ ] ,
"title" : "At a glance" ,
"type" : "row"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "thresholds"
} ,
"mappings" : [ ] ,
2022-02-08 12:22:06 +01:00
"min" : 0 ,
2021-08-19 09:57:53 +02:00
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 0
} ,
{
"color" : "yellow" ,
"value" : 1
} ,
{
"color" : "green" ,
"value" : 2
2022-02-08 12:22:06 +01:00
} ,
{
"color" : "#EAB839" ,
"value" : 120
} ,
{
"color" : "red" ,
"value" : 149
2021-08-19 09:57:53 +02:00
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 10 ,
2025-08-01 16:11:32 +05:30
"w" : 4 ,
2021-08-19 09:57:53 +02:00
"x" : 0 ,
"y" : 1
} ,
"id" : 52 ,
"options" : {
2024-08-21 17:10:29 +02:00
"minVizHeight" : 75 ,
"minVizWidth" : 75 ,
2022-02-08 12:22:06 +01:00
"orientation" : "auto" ,
2021-08-19 09:57:53 +02:00
"reduceOptions" : {
"calcs" : [
"lastNotNull"
] ,
"fields" : "" ,
"values" : false
} ,
"showThresholdLabels" : false ,
"showThresholdMarkers" : true ,
2025-08-01 16:11:32 +05:30
"sizing" : "auto"
2021-08-19 09:57:53 +02:00
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2021-08-19 09:57:53 +02:00
"exemplar" : true ,
2025-08-01 16:11:32 +05:30
"expr" : "libp2p_pubsub_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{fleet}}: {{datacenter}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
"title" : "Libp2p PubSub Peers" ,
"type" : "gauge"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2021-08-19 09:57:53 +02:00
} ,
"fieldConfig" : {
"defaults" : {
2022-02-08 12:22:06 +01:00
"color" : {
2025-08-01 16:11:32 +05:30
"mode" : "thresholds"
2022-02-08 12:22:06 +01:00
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
2025-08-01 16:11:32 +05:30
"color" : "blue" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2022-02-08 12:22:06 +01:00
}
]
} ,
2025-08-01 16:11:32 +05:30
"unit" : "dateTimeAsIso"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 10 ,
2025-08-01 16:11:32 +05:30
"w" : 7 ,
"x" : 4 ,
2022-09-20 10:01:54 +01:00
"y" : 1
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 46 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"colorMode" : "value" ,
"graphMode" : "none" ,
"justifyMode" : "auto" ,
"orientation" : "horizontal" ,
"percentChangeColorMode" : "standard" ,
"reduceOptions" : {
"calcs" : [
"lastNotNull"
] ,
"fields" : "" ,
"values" : false
2022-02-08 12:22:06 +01:00
} ,
2025-08-01 16:11:32 +05:30
"showPercentChange" : false ,
"text" : {
"titleSize" : 18 ,
"valueSize" : 20
} ,
"textMode" : "auto" ,
"wideLayout" : true
2021-08-19 09:57:53 +02:00
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-02-08 12:22:06 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2021-08-19 09:57:53 +02:00
"exemplar" : true ,
2025-08-01 16:11:32 +05:30
"expr" : "process_start_time_seconds{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} * 1000" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Node start times (UTC)" ,
"type" : "stat"
2022-02-08 12:22:06 +01:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2022-02-08 12:22:06 +01:00
"fieldConfig" : {
"defaults" : {
"color" : {
2025-08-01 16:11:32 +05:30
"mode" : "thresholds"
2022-02-08 12:22:06 +01:00
} ,
"mappings" : [ ] ,
2025-08-01 16:11:32 +05:30
"max" : 100 ,
"min" : 0 ,
2022-02-08 12:22:06 +01:00
"thresholds" : {
2025-08-01 16:11:32 +05:30
"mode" : "percentage" ,
2022-02-08 12:22:06 +01:00
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2022-02-08 12:22:06 +01:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "percent"
2022-02-08 12:22:06 +01:00
} ,
2022-09-20 10:01:54 +01:00
"overrides" : [ ]
2021-08-19 09:57:53 +02:00
} ,
2022-02-08 12:22:06 +01:00
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 10 ,
"w" : 8 ,
2025-08-01 16:11:32 +05:30
"x" : 11 ,
2022-09-20 10:01:54 +01:00
"y" : 1
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 58 ,
2022-02-08 12:22:06 +01:00
"options" : {
2025-08-01 16:11:32 +05:30
"displayMode" : "lcd" ,
2022-02-08 12:22:06 +01:00
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
2025-08-01 16:11:32 +05:30
"showLegend" : false
2022-02-08 12:22:06 +01:00
} ,
2025-08-01 16:11:32 +05:30
"maxVizHeight" : 300 ,
"minVizHeight" : 10 ,
"minVizWidth" : 0 ,
"namePlacement" : "auto" ,
"orientation" : "horizontal" ,
"reduceOptions" : {
"calcs" : [
"lastNotNull"
] ,
"fields" : "" ,
"values" : false
} ,
"showUnfilled" : true ,
"sizing" : "auto" ,
"text" : { } ,
"valueMode" : "color"
2022-02-08 12:22:06 +01:00
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2022-02-08 12:22:06 +01:00
"targets" : [
2021-08-19 09:57:53 +02:00
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "avg by (instance)(netdata_cpu_cpu_percentage_average{dimension=\"user\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2022-02-08 12:22:06 +01:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2022-02-08 12:22:06 +01:00
"refId" : "A"
2021-08-19 09:57:53 +02:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "CPU Usage" ,
"type" : "bargauge"
2022-02-08 12:22:06 +01:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "thresholds"
} ,
"custom" : {
"align" : "auto" ,
"cellOptions" : {
"type" : "auto"
} ,
"inspect" : false
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [
{
"matcher" : {
"id" : "byName" ,
"options" : "source"
} ,
"properties" : [
{
"id" : "custom.width" ,
"value" : 122
}
]
} ,
{
"matcher" : {
"id" : "byName" ,
"options" : "Time"
} ,
"properties" : [
{
"id" : "custom.width" ,
"value" : 181
}
]
}
]
2022-09-20 10:01:54 +01:00
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 10 ,
"w" : 5 ,
"x" : 19 ,
"y" : 1
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 85 ,
"options" : {
"cellHeight" : "sm" ,
"footer" : {
"countRows" : false ,
"fields" : "" ,
"reducer" : [
"sum"
] ,
"show" : false
} ,
"showHeader" : true ,
"sortBy" : [ ]
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2022-09-20 10:01:54 +01:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"editorMode" : "code" ,
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"format" : "table" ,
"instant" : true ,
"legendFormat" : "__auto" ,
"range" : false ,
2022-09-20 10:01:54 +01:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Version" ,
"transformations" : [
{
"id" : "filterFieldsByName" ,
"options" : {
"include" : {
"names" : [
"version" ,
"source" ,
"Time"
]
}
}
}
] ,
"type" : "table"
2022-09-20 10:01:54 +01:00
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2022-02-08 12:22:06 +01:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2022-02-08 12:22:06 +01:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2022-02-08 12:22:06 +01:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2022-02-08 12:22:06 +01:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2022-02-08 12:22:06 +01:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 9 ,
2022-02-08 12:22:06 +01:00
"w" : 12 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 11
2022-02-08 12:22:06 +01:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 81 ,
2022-02-08 12:22:06 +01:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2022-02-08 12:22:06 +01:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2022-09-20 10:01:54 +01:00
"mode" : "single" ,
"sort" : "none"
2022-02-08 12:22:06 +01:00
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2022-02-08 12:22:06 +01:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2022-09-20 10:01:54 +01:00
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"expr" : "libp2p_autonat_reachability_confidence{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", reachability=\"Reachable\"}" ,
"legendFormat" : "{{instance}}:{{reachability}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2022-02-08 12:22:06 +01:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Node is Reachable (Experimental)" ,
2022-02-08 12:22:06 +01:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2022-02-08 12:22:06 +01:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2022-02-08 12:22:06 +01:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2022-02-08 12:22:06 +01:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2022-02-08 12:22:06 +01:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2022-02-08 12:22:06 +01:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 9 ,
2022-02-08 12:22:06 +01:00
"w" : 12 ,
"x" : 12 ,
2025-08-01 16:11:32 +05:30
"y" : 11
2022-02-08 12:22:06 +01:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 82 ,
2022-02-08 12:22:06 +01:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2022-02-08 12:22:06 +01:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2022-09-20 10:01:54 +01:00
"mode" : "single" ,
"sort" : "none"
2022-02-08 12:22:06 +01:00
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2022-02-08 12:22:06 +01:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2022-09-20 10:01:54 +01:00
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"expr" : "libp2p_autonat_reachability_confidence{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", reachability=\"NotReachable\"}" ,
"legendFormat" : "{{instance}}:{{reachability}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2022-02-08 12:22:06 +01:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Node is NotReachable (Experimental)" ,
2022-02-08 12:22:06 +01:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2025-08-01 16:11:32 +05:30
"description" : "" ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
2025-08-01 16:11:32 +05:30
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"barWidthFactor" : 0.6 ,
"drawStyle" : "line" ,
"fillOpacity" : 10 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "never" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"links" : [ ] ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
} ,
"unit" : "binBps"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 9 ,
2021-08-19 09:57:53 +02:00
"w" : 12 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 20
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 78 ,
"interval" : "15s" ,
"maxDataPoints" : 1000 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"hideZeros" : false ,
"mode" : "single" ,
"sort" : "none"
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "rate(libp2p_network_bytes_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", direction=\"in\"}[$__rate_interval])" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}:{{direction}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Inbound Traffic" ,
"type" : "timeseries"
2021-08-19 09:57:53 +02:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"description" : "" ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
2025-08-01 16:11:32 +05:30
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"barWidthFactor" : 0.6 ,
"drawStyle" : "line" ,
"fillOpacity" : 10 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "never" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"links" : [ ] ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
} ,
"unit" : "binBps"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 9 ,
2021-08-19 09:57:53 +02:00
"w" : 12 ,
"x" : 12 ,
2025-08-01 16:11:32 +05:30
"y" : 20
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 79 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"hideZeros" : false ,
"mode" : "single" ,
"sort" : "none"
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "rate(libp2p_network_bytes_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", direction=\"out\"}[$__rate_interval])" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}:{{direction}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Outbound Traffic" ,
"type" : "timeseries"
2021-08-19 09:57:53 +02:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"description" : "" ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
2025-08-01 16:11:32 +05:30
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"barWidthFactor" : 0.6 ,
"drawStyle" : "line" ,
"fillOpacity" : 10 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "never" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"links" : [ ] ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
} ,
"unit" : "short"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 9 ,
2021-08-19 09:57:53 +02:00
"w" : 12 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 29
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 11 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"hideZeros" : false ,
"mode" : "single" ,
"sort" : "none"
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "((increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])))" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{type}}: {{instance}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Messages (1m rate)" ,
"type" : "timeseries"
2021-08-19 09:57:53 +02:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
2025-08-01 16:11:32 +05:30
"color" : {
"mode" : "palette-classic"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"barWidthFactor" : 0.6 ,
"drawStyle" : "line" ,
"fillOpacity" : 9 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 9 ,
2021-08-19 09:57:53 +02:00
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 12 ,
"y" : 29
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 54 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"tooltip" : {
"hideZeros" : false ,
"mode" : "single" ,
"sort" : "none"
2021-08-19 09:57:53 +02:00
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2022-02-08 12:22:06 +01:00
"targets" : [
2021-08-19 09:57:53 +02:00
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum by (type)((increase(waku_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "peer {{type}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2022-02-08 12:22:06 +01:00
"refId" : "A"
2022-09-20 10:01:54 +01:00
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum by (type)((increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))" ,
2022-09-20 10:01:54 +01:00
"hide" : false ,
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "store {{type}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
"refId" : "B"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum by (type)((increase(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))" ,
2022-09-20 10:01:54 +01:00
"hide" : false ,
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "node {{type}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
"refId" : "C"
2021-08-19 09:57:53 +02:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Waku Errors" ,
"type" : "timeseries"
2021-08-19 09:57:53 +02:00
} ,
2024-08-21 17:10:29 +02:00
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 8 ,
2024-08-21 17:10:29 +02:00
"w" : 12 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 38
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 66 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2024-08-21 17:10:29 +02:00
"mode" : "single" ,
"sort" : "none"
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "count(count by (contentTopic)((waku_node_messages_total or logos_delivery_node_messages_total)))" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "content topics" ,
2024-08-21 17:10:29 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Total Content Topics" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 5 ,
2024-08-21 17:10:29 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
2025-08-01 16:11:32 +05:30
"lineInterpolation" : "stepBefore" ,
"lineStyle" : {
"fill" : "solid"
} ,
"lineWidth" : 2 ,
2024-08-21 17:10:29 +02:00
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"value" : 0
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 8 ,
2024-08-21 17:10:29 +02:00
"w" : 12 ,
"x" : 12 ,
2025-08-01 16:11:32 +05:30
"y" : 38
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 68 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
2025-08-01 16:11:32 +05:30
"displayMode" : "table" ,
"placement" : "right" ,
2024-08-21 17:10:29 +02:00
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2024-08-21 17:10:29 +02:00
}
} ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"pluginVersion" : "12.1.0" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"editorMode" : "code" ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "{{instance}}:{{version}}" ,
2024-08-21 17:10:29 +02:00
"range" : true ,
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Waku version" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
2025-08-01 16:11:32 +05:30
{
"collapsed" : false ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 46
} ,
"id" : 17 ,
"panels" : [ ] ,
"title" : "General" ,
"type" : "row"
} ,
2021-08-19 09:57:53 +02:00
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"description" : "" ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
2021-08-19 09:57:53 +02:00
"w" : 12 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 47
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 48 ,
2021-08-19 09:57:53 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2022-09-20 10:01:54 +01:00
"mode" : "single" ,
"sort" : "none"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-02-08 12:22:06 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2021-08-19 09:57:53 +02:00
"exemplar" : true ,
2025-08-01 16:11:32 +05:30
"expr" : "waku_node_filters{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Waku Node Filters" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
2022-02-08 12:22:06 +01:00
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2022-02-08 12:22:06 +01:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2022-02-08 12:22:06 +01:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2022-02-08 12:22:06 +01:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2022-02-08 12:22:06 +01:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
2021-08-19 09:57:53 +02:00
"w" : 12 ,
"x" : 12 ,
2025-08-01 16:11:32 +05:30
"y" : 47
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 50 ,
2021-08-19 09:57:53 +02:00
"options" : {
2022-02-08 12:22:06 +01:00
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2022-02-08 12:22:06 +01:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2022-09-20 10:01:54 +01:00
"mode" : "single" ,
"sort" : "none"
2022-02-08 12:22:06 +01:00
}
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-02-08 12:22:06 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2021-08-19 09:57:53 +02:00
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{type}}: {{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Waku Node Errors" ,
2022-02-08 12:22:06 +01:00
"type" : "timeseries"
2021-08-19 09:57:53 +02:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
2025-08-01 16:11:32 +05:30
"color" : {
"mode" : "palette-classic"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"barWidthFactor" : 0.6 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 6 ,
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 0 ,
2024-08-21 17:10:29 +02:00
"y" : 53
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 60 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"hideZeros" : false ,
"mode" : "single" ,
"sort" : "none"
}
} ,
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "libp2p_pubsub_topics {instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "Topics: {{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
} ,
{
2025-08-01 16:11:32 +05:30
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "libp2p_pubsub_subscriptions_total {instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Subscriptions: {{instance}}" ,
"refId" : "B"
} ,
2022-09-20 10:01:54 +01:00
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "libp2p_pubsub_unsubscriptions_total {instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Unsubscriptions: {{instance}}" ,
"refId" : "C"
2022-09-20 10:01:54 +01:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Pubsub Topics" ,
"type" : "timeseries"
2021-08-19 09:57:53 +02:00
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 50 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2021-08-19 09:57:53 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
2025-08-01 16:11:32 +05:30
"mode" : "normal"
2021-08-19 09:57:53 +02:00
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2021-08-19 09:57:53 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "short"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 6 ,
2022-02-08 12:22:06 +01:00
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 12 ,
"y" : 53
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 8 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2021-08-19 09:57:53 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
2021-08-19 09:57:53 +02:00
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"expr" : "sum by (instance)(libp2p_pubsub_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "LibP2P PubSub Peers" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 50 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2021-08-19 09:57:53 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
2025-08-01 16:11:32 +05:30
"mode" : "normal"
2021-08-19 09:57:53 +02:00
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2021-08-19 09:57:53 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "short"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 6 ,
2022-02-08 12:22:06 +01:00
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 0 ,
"y" : 59
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 2 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2021-08-19 09:57:53 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
2021-08-19 09:57:53 +02:00
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"expr" : "sum by (instance)(libp2p_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "LibP2P Peers" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2022-02-08 12:22:06 +01:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2022-02-08 12:22:06 +01:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2022-02-08 12:22:06 +01:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 3 ,
2022-02-08 12:22:06 +01:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2022-02-08 12:22:06 +01:00
"lineInterpolation" : "linear" ,
2025-08-01 16:11:32 +05:30
"lineStyle" : {
"fill" : "solid"
} ,
2022-02-08 12:22:06 +01:00
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2022-02-08 12:22:06 +01:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2022-02-08 12:22:06 +01:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2022-02-08 12:22:06 +01:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "short"
2022-02-08 12:22:06 +01:00
} ,
2022-09-20 10:01:54 +01:00
"overrides" : [ ]
2022-02-08 12:22:06 +01:00
} ,
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 6 ,
2022-02-08 12:22:06 +01:00
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 12 ,
"y" : 59
2022-02-08 12:22:06 +01:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 83 ,
2022-02-08 12:22:06 +01:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2022-02-08 12:22:06 +01:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2022-02-08 12:22:06 +01:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2022-02-08 12:22:06 +01:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-02-08 12:22:06 +01:00
} ,
2022-09-20 10:01:54 +01:00
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2022-02-08 12:22:06 +01:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2022-09-20 10:01:54 +01:00
"range" : true ,
2022-02-08 12:22:06 +01:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Peer Store Size" ,
2022-02-08 12:22:06 +01:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 10 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2021-08-19 09:57:53 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2021-08-19 09:57:53 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "short"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 6 ,
2022-02-08 12:22:06 +01:00
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 0 ,
"y" : 65
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 3 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2021-08-19 09:57:53 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
2021-08-19 09:57:53 +02:00
"displayMode" : "list" ,
2025-08-01 16:11:32 +05:30
"placement" : "right" ,
2022-09-20 10:01:54 +01:00
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"expr" : "sum by (type)(libp2p_open_streams{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2022-09-20 10:01:54 +01:00
"legendFormat" : "{{type}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "LibP2P Open Streams" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
"fieldConfig" : {
"defaults" : {
2025-08-01 16:11:32 +05:30
"color" : {
"mode" : "palette-classic"
} ,
2022-09-20 10:01:54 +01:00
"custom" : {
2025-08-01 16:11:32 +05:30
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"barWidthFactor" : 0.6 ,
"drawStyle" : "line" ,
"fillOpacity" : 10 ,
"gradientMode" : "none" ,
2022-09-20 10:01:54 +01:00
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2025-08-01 16:11:32 +05:30
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
2022-09-20 10:01:54 +01:00
"scaleDistribution" : {
"type" : "linear"
2025-08-01 16:11:32 +05:30
} ,
"showPoints" : "never" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
2022-09-20 10:01:54 +01:00
}
2025-08-01 16:11:32 +05:30
} ,
"links" : [ ] ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2025-08-01 16:11:32 +05:30
} ,
{
"color" : "red" ,
"value" : 80
}
]
} ,
"unit" : "short"
2022-09-20 10:01:54 +01:00
} ,
"overrides" : [ ]
} ,
2021-08-19 09:57:53 +02:00
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
2022-09-20 10:01:54 +01:00
"w" : 12 ,
2025-08-01 16:11:32 +05:30
"x" : 12 ,
"y" : 65
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 9 ,
2022-09-20 10:01:54 +01:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2022-09-20 10:01:54 +01:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
"displayMode" : "table" ,
"placement" : "right" ,
"showLegend" : true
2022-09-20 10:01:54 +01:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2022-09-20 10:01:54 +01:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2022-09-20 10:01:54 +01:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "libp2p_pubsub_validation_success_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"hide" : false ,
2022-09-20 10:01:54 +01:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "success {{instance}}" ,
2022-09-20 10:01:54 +01:00
"refId" : "A"
} ,
2021-08-19 09:57:53 +02:00
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "libp2p_pubsub_validation_failure_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
2021-08-19 09:57:53 +02:00
"hide" : false ,
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "failure {{instance}}" ,
2022-09-20 10:01:54 +01:00
"refId" : "B"
2025-08-01 16:11:32 +05:30
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "libp2p_pubsub_validation_ignore_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "ignore {{instance}}" ,
"refId" : "C"
2022-09-20 10:01:54 +01:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "LibP2P Validations" ,
"type" : "timeseries"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 10 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2021-08-19 09:57:53 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2021-08-19 09:57:53 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"unit" : "short"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
"w" : 12 ,
2022-09-20 10:01:54 +01:00
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 71
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 6 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2021-08-19 09:57:53 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
2021-08-19 09:57:53 +02:00
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"expr" : "sum by (instance)(process_open_fds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Open File Descriptors" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 10 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2021-08-19 09:57:53 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2021-08-19 09:57:53 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"unit" : "short"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
"w" : 12 ,
"x" : 12 ,
"y" : 71
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 7 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2021-08-19 09:57:53 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
2021-08-19 09:57:53 +02:00
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "libp2p_total_dial_attempts_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"format" : "time_series" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Attempts: {{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
2025-08-01 16:11:32 +05:30
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "libp2p_failed_dials_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Failed: {{instance}}" ,
"refId" : "B"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "libp2p_successful_dials_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Successful: {{instance}}" ,
"refId" : "C"
2021-08-19 09:57:53 +02:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "LibP2P Dials" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"description" : "" ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 4 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"unit" : "decbytes"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 77
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 44 ,
2021-08-19 09:57:53 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2022-09-20 10:01:54 +01:00
"mode" : "single" ,
"sort" : "none"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "nim_gc_mem_bytes{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"interval" : "" ,
"legendFormat" : "Nim total memory: {{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
2025-08-01 16:11:32 +05:30
} ,
2022-09-20 10:01:54 +01:00
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2025-08-01 16:11:32 +05:30
"exemplar" : true ,
"expr" : "nim_gc_mem_occupied_bytes{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Nim occupied memory: {{instance}}" ,
"refId" : "B"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "nim_gc_heap_instance_occupied_summed_bytes{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "Nim total heap: {{instance}}" ,
"refId" : "C"
2022-09-20 10:01:54 +01:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Nim Memory Usage" ,
"type" : "timeseries"
2022-09-20 10:01:54 +01:00
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 10 ,
2021-08-19 09:57:53 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2021-08-19 09:57:53 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2021-08-19 09:57:53 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "percent"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
"w" : 12 ,
"x" : 12 ,
"y" : 77
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 10 ,
2021-08-19 09:57:53 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2021-08-19 09:57:53 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"max"
] ,
2021-08-19 09:57:53 +02:00
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"exemplar" : true ,
2025-08-01 16:11:32 +05:30
"expr" : "avg by (instance)(netdata_cpu_cpu_percentage_average{dimension=\"user\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2024-08-21 17:10:29 +02:00
"hide" : false ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "CPU Usage" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
} ,
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2021-08-19 09:57:53 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
2024-08-21 17:10:29 +02:00
"axisBorderShow" : false ,
2022-09-20 10:01:54 +01:00
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
2021-08-19 09:57:53 +02:00
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2021-08-19 09:57:53 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
2024-08-21 17:10:29 +02:00
"insertNulls" : false ,
2021-08-19 09:57:53 +02:00
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2021-08-19 09:57:53 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "decbytes"
2021-08-19 09:57:53 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2022-09-20 10:01:54 +01:00
"h" : 8 ,
2025-08-01 16:11:32 +05:30
"w" : 12 ,
"x" : 12 ,
"y" : 83
2021-08-19 09:57:53 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 64 ,
2021-08-19 09:57:53 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
2022-09-20 10:01:54 +01:00
"placement" : "bottom" ,
"showLegend" : true
2021-08-19 09:57:53 +02:00
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2022-09-20 10:01:54 +01:00
"mode" : "single" ,
"sort" : "none"
2021-08-19 09:57:53 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2021-08-19 09:57:53 +02:00
"targets" : [
{
2022-09-20 10:01:54 +01:00
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2022-09-20 10:01:54 +01:00
} ,
2021-08-19 09:57:53 +02:00
"exemplar" : true ,
2025-08-01 16:11:32 +05:30
"expr" : "nim_gc_heap_instance_occupied_bytes{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
2021-08-19 09:57:53 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}} {{type_name}}" ,
2021-08-19 09:57:53 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Heap allocation" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 10 ,
2024-08-21 17:10:29 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2024-08-21 17:10:29 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2024-08-21 17:10:29 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
} ,
"unit" : "decbytes"
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
"w" : 12 ,
"x" : 0 ,
"y" : 85
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 4 ,
2024-08-21 17:10:29 +02:00
"options" : {
2025-08-01 16:11:32 +05:30
"alertThreshold" : true ,
2024-08-21 17:10:29 +02:00
"legend" : {
2025-08-01 16:11:32 +05:30
"calcs" : [
"mean"
] ,
2024-08-21 17:10:29 +02:00
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
2024-08-21 17:10:29 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"expr" : "sum by (instance)(process_virtual_memory_bytes{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2024-08-21 17:10:29 +02:00
"interval" : "" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
2024-08-21 17:10:29 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Virtual Memory" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
2025-08-01 16:11:32 +05:30
"fillOpacity" : 10 ,
2024-08-21 17:10:29 +02:00
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
2025-08-01 16:11:32 +05:30
"showPoints" : "never" ,
2024-08-21 17:10:29 +02:00
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
2025-08-01 16:11:32 +05:30
"links" : [ ] ,
2024-08-21 17:10:29 +02:00
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
} ,
2025-08-01 16:11:32 +05:30
"unit" : "decbytes"
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 6 ,
"w" : 12 ,
"x" : 12 ,
"y" : 91
} ,
"id" : 5 ,
"options" : {
"alertThreshold" : true ,
"legend" : {
"calcs" : [
"mean"
] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"hideZeros" : false ,
"mode" : "multi" ,
"sort" : "desc"
}
} ,
"pluginVersion" : "11.5.2" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"expr" : "sum by (instance)(process_resident_memory_bytes{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
"refId" : "A"
}
] ,
"title" : "Resident Memory" ,
"type" : "timeseries"
} ,
{
"collapsed" : true ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 97
} ,
"id" : 72 ,
"panels" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 3 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 122
} ,
"id" : 70 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "increase(waku_bridge_transfers_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m])" ,
"interval" : "" ,
"legendFormat" : "{{fleet}} : {{type}}" ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "increase(envelopes_valid_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m])" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{fleet}} : v1_envelopes" ,
"refId" : "B"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(increase(waku_node_messages_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m]) or increase(logos_delivery_node_messages_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m]))" ,
2025-08-01 16:11:32 +05:30
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{fleet}} : v2_messages" ,
"refId" : "C"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "increase(envelopes_dropped_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m])" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{fleet}} : v1_envelopes_dropped ({{reason}})" ,
"refId" : "D"
}
] ,
"title" : "Bridge (10m rate)" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 122
} ,
"id" : 74 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "multi" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "connected_peers{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"interval" : "" ,
"legendFormat" : "v1_connected_peers" ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
"expr" : "libp2p_pubsub_peers{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "v2_connected_peers" ,
"refId" : "B"
}
] ,
"title" : "Connected Peers" ,
"type" : "timeseries"
}
] ,
"title" : "Bridge" ,
"type" : "row"
} ,
{
"collapsed" : true ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 98
} ,
"id" : 34 ,
"panels" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 6 ,
"w" : 12 ,
"x" : 0 ,
"y" : 99
} ,
"id" : 36 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
"refId" : "A"
}
] ,
"title" : "Waku Store Peers" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 2 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 6 ,
"w" : 12 ,
"x" : 12 ,
"y" : 99
} ,
"id" : 38 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : false
} ,
"tooltip" : {
"mode" : "multi" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
"expr" : "waku_store_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{type}}: {{instance}}" ,
"range" : true ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"hide" : false ,
"legendFormat" : "{{type}}: {{instance}}" ,
"range" : true ,
"refId" : "B"
}
] ,
"title" : "Waku Archive Messages" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 3 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 6 ,
"w" : 12 ,
"x" : 0 ,
"y" : 105
} ,
"id" : 62 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(increase(waku_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Store Queries (1m rate)" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 6 ,
"w" : 12 ,
"x" : 12 ,
"y" : 105
} ,
"id" : 40 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum by (type)((increase(waku_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m])))" ,
2025-08-01 16:11:32 +05:30
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{type}}" ,
"range" : true ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum by (type)((increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m])))" ,
2025-08-01 16:11:32 +05:30
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{type}}" ,
"range" : true ,
"refId" : "B"
}
] ,
"title" : "Waku Archive Errors (1m rate)" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"custom" : {
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"scaleDistribution" : {
"type" : "linear"
}
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 7 ,
"w" : 12 ,
"x" : 0 ,
"y" : 111
} ,
"id" : 77 ,
"maxDataPoints" : 60 ,
"options" : {
"calculate" : false ,
"calculation" : { } ,
"cellGap" : 2 ,
"cellValues" : { } ,
"color" : {
"exponent" : 0.5 ,
"fill" : "#b4ff00" ,
"mode" : "scheme" ,
"reverse" : false ,
"scale" : "exponential" ,
"scheme" : "RdYlGn" ,
"steps" : 128
} ,
"exemplars" : {
"color" : "rgba(255,0,255,0.7)"
} ,
"filterValues" : {
"le" : 1e-9
} ,
"legend" : {
"show" : false
} ,
"rowsFrame" : {
"layout" : "auto"
} ,
"showValue" : "never" ,
"tooltip" : {
"show" : true ,
"yHistogram" : true
} ,
"yAxis" : {
"axisPlacement" : "left" ,
"decimals" : 0 ,
"reverse" : false ,
"unit" : "s"
}
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum((increase(waku_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"hide" : false ,
"legendFormat" : "{{le}}" ,
"range" : true ,
"refId" : "B"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"expr" : "sum(increase(waku_store_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)" ,
"format" : "heatmap" ,
"hide" : true ,
"legendFormat" : "{{le}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Archive Query Duration" ,
"type" : "heatmap"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"custom" : {
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"scaleDistribution" : {
"type" : "linear"
}
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 7 ,
"w" : 12 ,
"x" : 12 ,
"y" : 111
} ,
"id" : 75 ,
"maxDataPoints" : 60 ,
"options" : {
"calculate" : false ,
"calculation" : { } ,
"cellGap" : 2 ,
"cellValues" : { } ,
"color" : {
"exponent" : 0.5 ,
"fill" : "#b4ff00" ,
"mode" : "scheme" ,
"reverse" : false ,
"scale" : "exponential" ,
"scheme" : "RdYlGn" ,
"steps" : 128
} ,
"exemplars" : {
"color" : "rgba(255,0,255,0.7)"
} ,
"filterValues" : {
"le" : 1e-9
} ,
"legend" : {
"show" : false
} ,
"rowsFrame" : {
"layout" : "auto"
} ,
"showValue" : "never" ,
"tooltip" : {
"show" : true ,
"yHistogram" : true
} ,
"yAxis" : {
"axisPlacement" : "left" ,
"decimals" : 0 ,
"reverse" : false ,
"unit" : "s"
}
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum((increase(waku_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"hide" : false ,
"instant" : false ,
"interval" : "" ,
"intervalFactor" : 1 ,
"legendFormat" : "{{le}}" ,
"refId" : "B"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
"expr" : "sum(increase(waku_store_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)" ,
"format" : "heatmap" ,
"hide" : true ,
"instant" : false ,
"interval" : "" ,
"intervalFactor" : 1 ,
"legendFormat" : "{{le}}" ,
"refId" : "A"
}
] ,
"title" : "Waku Archive Insert Duration" ,
"type" : "heatmap"
}
] ,
"title" : "Store/Archive" ,
"type" : "row"
} ,
{
"collapsed" : true ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 99
} ,
"id" : 87 ,
"panels" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 8 ,
"x" : 0 ,
"y" : 135
} ,
"id" : 93 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{type}} : {{instance}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Filter Requests" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 8 ,
"x" : 8 ,
"y" : 135
} ,
"id" : 89 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{instance}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Filter Subscriptions" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 8 ,
"x" : 16 ,
"y" : 135
} ,
"id" : 91 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{type}} : {{instance}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Filter Errors" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"custom" : {
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"scaleDistribution" : {
"type" : "linear"
}
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 143
} ,
"id" : 95 ,
"options" : {
"calculate" : false ,
"cellGap" : 2 ,
"color" : {
"exponent" : 0.5 ,
"fill" : "dark-orange" ,
"mode" : "scheme" ,
"reverse" : false ,
"scale" : "exponential" ,
"scheme" : "RdYlGn" ,
"steps" : 128
} ,
"exemplars" : {
"color" : "rgba(255,0,255,0.7)"
} ,
"filterValues" : {
"le" : 1e-9
} ,
"legend" : {
"show" : false
} ,
"rowsFrame" : {
"layout" : "auto"
} ,
"tooltip" : {
"show" : true ,
"yHistogram" : true
} ,
"yAxis" : {
"axisPlacement" : "left" ,
"decimals" : 0 ,
"reverse" : false ,
"unit" : "s"
}
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum((increase(waku_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"interval" : "" ,
"legendFormat" : "{{le}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Filter Request Duration" ,
"type" : "heatmap"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"custom" : {
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"scaleDistribution" : {
"type" : "linear"
}
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 143
} ,
"id" : 97 ,
"options" : {
"calculate" : false ,
"cellGap" : 2 ,
"color" : {
"exponent" : 0.5 ,
"fill" : "dark-orange" ,
"mode" : "scheme" ,
"reverse" : false ,
"scale" : "exponential" ,
"scheme" : "RdYlGn" ,
"steps" : 128
} ,
"exemplars" : {
"color" : "rgba(255,0,255,0.7)"
} ,
"filterValues" : {
"le" : 1e-9
} ,
"legend" : {
"show" : false
} ,
"rowsFrame" : {
"layout" : "auto"
} ,
"tooltip" : {
"show" : true ,
"yHistogram" : false
} ,
"yAxis" : {
"axisPlacement" : "left" ,
"decimals" : 0 ,
"reverse" : false ,
"unit" : "s"
}
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "sum((increase(waku_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"legendFormat" : "{{le}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Waku Filter Handle Message Duration" ,
"type" : "heatmap"
}
] ,
"title" : "Filter" ,
"type" : "row"
} ,
{
"collapsed" : true ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 100
} ,
"id" : 20 ,
"panels" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"description" : "Waku Filter Peers" ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 149
} ,
"id" : 22 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_filter_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"hide" : false ,
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
"refId" : "A"
}
] ,
"title" : "Waku Filter Peers" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"description" : "" ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 149
} ,
"id" : 26 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "{{type}}: {{instance}}" ,
"range" : true ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"expr" : "waku_legacy_filter_errors{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"legendFormat" : "{{type}} : {{instance}}" ,
"range" : true ,
"refId" : "B"
}
] ,
"title" : "Waku Filter Errors" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 157
} ,
"id" : 99 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"expr" : "waku_filter_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"legendFormat" : "{{type}} : {{instance}}" ,
"range" : true ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"expr" : "waku_legacy_filter_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"legendFormat" : "{{type}} : {{instance}}" ,
"range" : true ,
"refId" : "B"
}
] ,
"title" : "Waku Filter Messages" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 3 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 7 ,
"w" : 12 ,
"x" : 12 ,
"y" : 157
} ,
"id" : 24 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : true ,
"expr" : "waku_filter_subscribers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
"range" : true ,
"refId" : "A"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"expr" : "waku_legacy_filter_subscribers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}" ,
"hide" : false ,
"legendFormat" : "{{instance}}" ,
"range" : true ,
"refId" : "B"
}
] ,
"title" : "Waku Filter Subscribers" ,
"type" : "timeseries"
}
] ,
"title" : "Filter (Legacy)" ,
"type" : "row"
} ,
{
"collapsed" : true ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
2024-08-21 17:10:29 +02:00
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 101
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 28 ,
"panels" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 129
} ,
"id" : 30 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "{{instance}}" ,
"refId" : "A"
}
] ,
"title" : "Waku Lightpush Peers" ,
"type" : "timeseries"
2024-08-21 17:10:29 +02:00
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 129
} ,
"id" : 32 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"exemplar" : true ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"interval" : "" ,
"legendFormat" : "{{type}}: {[instance}}" ,
"refId" : "A"
}
] ,
"title" : "Waku Lightpush Errors" ,
"type" : "timeseries"
2024-08-21 17:10:29 +02:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Lightpush" ,
"type" : "row"
2024-08-21 17:10:29 +02:00
} ,
{
2025-08-01 16:11:32 +05:30
"collapsed" : true ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
"y" : 102
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 15 ,
"panels" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
}
]
} ,
"unit" : "none"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 122
} ,
"id" : 103 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "right" ,
"showLegend" : true
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
}
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "((increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])))/60" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"instant" : false ,
"legendFormat" : "{{fleet}}_{{datacenter}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Messages/second" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
}
]
} ,
"unit" : "deckbytes"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 122
} ,
"id" : 102 ,
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "right" ,
"showLegend" : true
} ,
"tooltip" : {
"mode" : "single" ,
"sort" : "none"
2024-08-21 17:10:29 +02:00
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"editorMode" : "code" ,
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})/(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"instant" : false ,
"legendFormat" : "{{fleet}}_{{datacenter}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Average msg size (kBytes)" ,
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "thresholds"
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
}
]
} ,
"unit" : "percent"
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 24 ,
"x" : 0 ,
"y" : 130
} ,
"id" : 101 ,
"options" : {
"displayMode" : "gradient" ,
"minVizHeight" : 10 ,
"minVizWidth" : 0 ,
"orientation" : "auto" ,
"reduceOptions" : {
"calcs" : [
"lastNotNull"
] ,
"fields" : "" ,
"values" : false
} ,
"showUnfilled" : true
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(rate(waku_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]) or rate(logos_delivery_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]))/scalar((rate(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]) or rate(logos_delivery_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h])))*100" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"instant" : false ,
"legendFormat" : "{{le}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Message distrubution %/kBytes (Last Hour)" ,
"type" : "bargauge"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "thresholds"
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
}
]
} ,
"unit" : "none"
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 138
} ,
"id" : 105 ,
"options" : {
"colorMode" : "value" ,
"graphMode" : "none" ,
"justifyMode" : "auto" ,
"orientation" : "auto" ,
"reduceOptions" : {
"calcs" : [
"lastNotNull"
] ,
"fields" : "" ,
"values" : false
} ,
"text" : {
"titleSize" : 15 ,
"valueSize" : 50
} ,
"textMode" : "auto"
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"instant" : false ,
"legendFormat" : "Direction:{{direction}} {{protocol}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Connected Peers per Protocol" ,
"type" : "stat"
2024-08-21 17:10:29 +02:00
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "thresholds"
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
"color" : "green"
}
]
} ,
"unit" : "none"
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 138
} ,
"id" : 104 ,
"options" : {
"colorMode" : "value" ,
"graphMode" : "none" ,
"justifyMode" : "auto" ,
"orientation" : "auto" ,
"reduceOptions" : {
"calcs" : [
"lastNotNull"
] ,
"fields" : "" ,
"values" : false
} ,
"text" : {
"titleSize" : 15 ,
"valueSize" : 50
} ,
"textMode" : "auto"
} ,
"pluginVersion" : "9.2.5" ,
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"editorMode" : "code" ,
"exemplar" : false ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"format" : "heatmap" ,
"instant" : false ,
"legendFormat" : "Direction:{{direction}} {{protocol}}" ,
"range" : true ,
"refId" : "A"
}
] ,
"title" : "Connected Streams per Protocol" ,
"type" : "stat"
2024-08-21 17:10:29 +02:00
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Messages" ,
"type" : "row"
2024-08-21 17:10:29 +02:00
} ,
{
"collapsed" : false ,
"gridPos" : {
"h" : 1 ,
"w" : 24 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 103
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 112 ,
2024-08-21 17:10:29 +02:00
"panels" : [ ] ,
2025-08-01 16:11:32 +05:30
"title" : "RLN" ,
2024-08-21 17:10:29 +02:00
"type" : "row"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 104
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 107 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2024-08-21 17:10:29 +02:00
"mode" : "single" ,
"sort" : "none"
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{type}}_{{datacenter}}" ,
"range" : true ,
2024-08-21 17:10:29 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Invalid Messages" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
}
} ,
"overrides" : [ ]
} ,
"gridPos" : {
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
2025-08-01 16:11:32 +05:30
"y" : 104
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 110 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2024-08-21 17:10:29 +02:00
"mode" : "single" ,
"sort" : "none"
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_rln_valid_messages_total_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_valid_messages_total_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{__name__}}_{{datacenter}}" ,
"range" : true ,
2024-08-21 17:10:29 +02:00
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Valid RLN Messages" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 8 ,
"w" : 12 ,
2024-08-21 17:10:29 +02:00
"x" : 0 ,
2025-08-01 16:11:32 +05:30
"y" : 112
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 109 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2024-08-21 17:10:29 +02:00
"mode" : "single" ,
"sort" : "none"
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{__name__}}_{{datacenter}}" ,
2024-08-21 17:10:29 +02:00
"range" : true ,
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Spam messages" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 8 ,
"w" : 12 ,
"x" : 12 ,
"y" : 112
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 108 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2024-08-21 17:10:29 +02:00
"mode" : "single" ,
"sort" : "none"
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{__name__}}_{{datacenter}}" ,
2024-08-21 17:10:29 +02:00
"range" : true ,
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "Proof Verification (s)" ,
2024-08-21 17:10:29 +02:00
"type" : "timeseries"
} ,
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"fieldConfig" : {
"defaults" : {
"color" : {
"mode" : "palette-classic"
} ,
"custom" : {
"axisBorderShow" : false ,
"axisCenteredZero" : false ,
"axisColorMode" : "text" ,
"axisLabel" : "" ,
"axisPlacement" : "auto" ,
"barAlignment" : 0 ,
2025-08-01 16:11:32 +05:30
"barWidthFactor" : 0.6 ,
2024-08-21 17:10:29 +02:00
"drawStyle" : "line" ,
"fillOpacity" : 0 ,
"gradientMode" : "none" ,
"hideFrom" : {
"legend" : false ,
"tooltip" : false ,
"viz" : false
} ,
"insertNulls" : false ,
"lineInterpolation" : "linear" ,
"lineWidth" : 1 ,
"pointSize" : 5 ,
"scaleDistribution" : {
"type" : "linear"
} ,
"showPoints" : "auto" ,
"spanNulls" : false ,
"stacking" : {
"group" : "A" ,
"mode" : "none"
} ,
"thresholdsStyle" : {
"mode" : "off"
}
} ,
"mappings" : [ ] ,
"thresholds" : {
"mode" : "absolute" ,
"steps" : [
{
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"color" : "green"
2024-08-21 17:10:29 +02:00
} ,
{
"color" : "red" ,
"value" : 80
}
]
2025-08-01 16:11:32 +05:30
}
2024-08-21 17:10:29 +02:00
} ,
"overrides" : [ ]
} ,
"gridPos" : {
2025-08-01 16:11:32 +05:30
"h" : 8 ,
"w" : 12 ,
"x" : 0 ,
"y" : 120
2024-08-21 17:10:29 +02:00
} ,
2025-08-01 16:11:32 +05:30
"id" : 113 ,
2024-08-21 17:10:29 +02:00
"options" : {
"legend" : {
"calcs" : [ ] ,
"displayMode" : "list" ,
"placement" : "bottom" ,
"showLegend" : true
} ,
"tooltip" : {
2025-08-01 16:11:32 +05:30
"hideZeros" : false ,
2024-08-21 17:10:29 +02:00
"mode" : "single" ,
"sort" : "none"
}
} ,
2025-08-01 16:11:32 +05:30
"pluginVersion" : "11.5.2" ,
2024-08-21 17:10:29 +02:00
"targets" : [
{
"datasource" : {
"type" : "prometheus" ,
2025-08-01 16:11:32 +05:30
"uid" : "P6693426190CB2316"
2024-08-21 17:10:29 +02:00
} ,
"editorMode" : "code" ,
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_
Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).
Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.
rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.
library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.
BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.
* refactor(metrics): prefix auxiliary app metrics with logos_delivery_
Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).
These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.
* chore(metrics): query old and new metric names in Grafana dashboards
212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:
sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))
The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.
Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.
Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.
Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.
* test(e2e): expect logos_delivery_-prefixed metric names
The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.
* docs(operators): document the logos_delivery_ metric prefix
Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.
* refactor(metrics): name the store fleet metrics after store, not relay
logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.
The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.
Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00
"expr" : "(waku_rln_number_registered_memberships{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_number_registered_memberships{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})" ,
2025-08-01 16:11:32 +05:30
"legendFormat" : "{{__name__}}_{{datacenter}}" ,
2024-08-21 17:10:29 +02:00
"range" : true ,
"refId" : "A"
}
] ,
2025-08-01 16:11:32 +05:30
"title" : "RLN Memberships" ,
2021-08-19 09:57:53 +02:00
"type" : "timeseries"
}
] ,
2025-08-01 16:11:32 +05:30
"preload" : false ,
"refresh" : "30s" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"schemaVersion" : 41 ,
2021-08-19 09:57:53 +02:00
"tags" : [ ] ,
"templating" : {
2025-08-01 16:11:32 +05:30
"list" : [
{
"current" : {
"text" : "node-.*" ,
"value" : "node-.*"
} ,
"includeAll" : false ,
"label" : "Hostname regex" ,
"name" : "host" ,
"options" : [
{
"selected" : true ,
"text" : "node-.*" ,
"value" : "node-.*"
}
] ,
"query" : "node-.*" ,
"type" : "custom"
} ,
{
"current" : {
"text" : [
"status.prod"
] ,
"value" : [
"status.prod"
]
} ,
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"definition" : "label_values(libp2p_peers, fleet)" ,
"includeAll" : false ,
"label" : "Fleet name" ,
"multi" : true ,
"name" : "fleet" ,
"options" : [ ] ,
"query" : {
"query" : "label_values(libp2p_peers, fleet)" ,
"refId" : "StandardVariableQuery"
} ,
"refresh" : 1 ,
"regex" : "/waku|status|shards/" ,
"type" : "query"
} ,
{
"current" : {
"text" : "All" ,
"value" : "$__all"
} ,
"datasource" : {
"type" : "prometheus" ,
"uid" : "P6693426190CB2316"
} ,
"definition" : "label_values(libp2p_peers, datacenter)" ,
"includeAll" : true ,
"label" : "Data Center" ,
"multi" : true ,
"name" : "dc" ,
"options" : [ ] ,
"query" : {
"query" : "label_values(libp2p_peers, datacenter)" ,
"refId" : "StandardVariableQuery"
} ,
"refresh" : 1 ,
"regex" : "" ,
"sort" : 1 ,
"type" : "query"
}
]
2021-08-19 09:57:53 +02:00
} ,
"time" : {
2025-08-01 16:11:32 +05:30
"from" : "now-10d" ,
"to" : "now"
2021-08-19 09:57:53 +02:00
} ,
"timepicker" : {
"refresh_intervals" : [
"10s" ,
"30s" ,
"1m" ,
"5m" ,
"15m" ,
"30m" ,
"1h" ,
"2h" ,
"1d"
]
} ,
2022-09-20 10:01:54 +01:00
"timezone" : "browser" ,
2025-08-01 16:11:32 +05:30
"title" : "Nim-Waku Single Node" ,
"uid" : "TgQ8wf8Vz" ,
chore(metrics): sync fleet dashboards from Grafana (#4079)
Refreshed the two shared-Grafana dashboards and re-applied the old-or-new metric
name union from #4074, so the panels keep working across the rename.
waku-fleet-dashboard.json v181 -> v191
drops the five deprecated Store v2 panels; the "Store V3 Times wire-to-wire"
query moves to sum without (phase) and changes() instead of deriv(), which is
the better staleness filter for a gauge that moves and returns
waku-single-node-dashboard.json v7 -> v9
no functional change, a Grafana 12 re-save (pluginVersion, threshold base,
schemaVersion 40 -> 41)
The $dc variable is populated from libp2p_peers rather than waku_version as the
export had it: libp2p_* comes from a dependency and is immune to our renames, so
it cannot rot the next time metrics are renamed. Grafana keeps variable queries
outside `expr`, so the #4074 union pass never saw this one.
Store/Archive is collapsed again, matching the other rows; the export had it open
because of whoever last hit save. "Store v3 query request rates" moves to x=0 to
fill the space left by deleting its v2 sibling.
The dashboards under apps/ are deliberately not synced. They are provisioned into
the docker-compose stacks that ship beside them (docker-compose.yml mounts
monitoring/configuration/dashboards into /var/lib/grafana/dashboards), so they
must match the local datasources.yaml and prometheus-config.yml, not status.prod.
The Grafana copies carry a datasource uid that the local stack does not provision
and job selectors that its single scrape job cannot match.
2026-07-30 16:42:39 +01:00
"version" : 9
2021-08-19 09:57:53 +02:00
}
