logos-messaging/logos-messaging-nim
1
0
Fork 0
mirror of https://github.com/logos-messaging/logos-messaging-nim.git synced 2026-07-29 19:43:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(metrics): give every metric a logos_delivery_ prefix (#4074)

Browse Source 
* refactor(metrics): prefix node metrics with logos_delivery_

Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).

Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.

rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.

library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.

BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.

* refactor(metrics): prefix auxiliary app metrics with logos_delivery_

Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).

These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.

* chore(metrics): query old and new metric names in Grafana dashboards

212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:

  sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
              or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))

The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.

Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.

Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.

Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.

* test(e2e): expect logos_delivery_-prefixed metric names

The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.

* docs(operators): document the logos_delivery_ metric prefix

Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.

* refactor(metrics): name the store fleet metrics after store, not relay

logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.

The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.

Note that both metrics are wrong independently of their name, see the PR
description.
This commit is contained in:
Igor Sirotin 2026-07-29 18:56:50 +01:00 committed by GitHub
parent 26e1d62524
commit 8125cd0262
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
66 changed files with 769 additions and 655 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
apps/chat2bridge/chat2bridge.nim
View File

@ -31,9 +31,10 @@ import
# Common cli config
./config_chat2bridge
declarePublicCounter chat2_mb_transfers,
declarePublicCounter logos_delivery_chat2_mb_transfers,
"Number of messages transferred between chat2 and Matterbridge", ["type"]
declarePublicCounter chat2_mb_dropped, "Number of messages dropped", ["reason"]
declarePublicCounter logos_delivery_chat2_mb_dropped,
"Number of messages dropped", ["reason"]
logScope:
topics = "chat2bridge"
@ -96,12 +97,12 @@ proc toChat2(cmb: Chat2MatterBridge, jsonNode: JsonNode) {.async.} =
if cmb.seen.containsOrAdd(msg.payload.hash()):
# This is a duplicate message. Return.
chat2_mb_dropped.inc(labelValues = ["duplicate"])
logos_delivery_chat2_mb_dropped.inc(labelValues = ["duplicate"])
return
trace "Post Matterbridge message to chat2"
chat2_mb_transfers.inc(labelValues = ["mb_to_chat2"])
logos_delivery_chat2_mb_transfers.inc(labelValues = ["mb_to_chat2"])
(await cmb.nodev2.publish(Opt.some(DefaultPubsubTopic), msg)).isOkOr:
error "failed to publish message", error = error
@ -111,17 +112,17 @@ proc toMatterbridge(
) {.gcsafe, raises: [Exception].} =
if cmb.seen.containsOrAdd(msg.payload.hash()):
# This is a duplicate message. Return.
chat2_mb_dropped.inc(labelValues = ["duplicate"])
logos_delivery_chat2_mb_dropped.inc(labelValues = ["duplicate"])
return
if msg.contentTopic != cmb.contentTopic:
# Only bridge messages on the configured content topic
chat2_mb_dropped.inc(labelValues = ["filtered"])
logos_delivery_chat2_mb_dropped.inc(labelValues = ["filtered"])
return
trace "Post chat2 message to Matterbridge"
chat2_mb_transfers.inc(labelValues = ["chat2_to_mb"])
logos_delivery_chat2_mb_transfers.inc(labelValues = ["chat2_to_mb"])
let chat2Msg = Chat2Message.init(msg.payload)
@ -132,7 +133,7 @@ proc toMatterbridge(
text = string.fromBytes(chat2Msg[].payload), username = chat2Msg[].nick
)
.containsValue(true):
chat2_mb_dropped.inc(labelValues = ["duplicate"])
logos_delivery_chat2_mb_dropped.inc(labelValues = ["duplicate"])
error "Matterbridge host unreachable. Dropping message."
proc pollMatterbridge(cmb: Chat2MatterBridge, handler: MbMessageHandler) {.async.} =

39
apps/liteprotocoltester/lpt_metrics.nim
View File

@ -5,50 +5,55 @@ import metrics
export metrics
declarePublicGauge lpt_receiver_sender_peer_count, "count of sender peers"
declarePublicGauge logos_delivery_lpt_receiver_sender_peer_count,
"count of sender peers"
declarePublicCounter lpt_receiver_received_messages_count,
declarePublicCounter logos_delivery_lpt_receiver_received_messages_count,
"number of messages received per peer", ["peer"]
declarePublicCounter lpt_receiver_received_bytes,
declarePublicCounter logos_delivery_lpt_receiver_received_bytes,
"number of received bytes per peer", ["peer"]
declarePublicGauge lpt_receiver_missing_messages_count,
declarePublicGauge logos_delivery_lpt_receiver_missing_messages_count,
"number of missing messages per peer", ["peer"]
declarePublicCounter lpt_receiver_duplicate_messages_count,
declarePublicCounter logos_delivery_lpt_receiver_duplicate_messages_count,
"number of duplicate messages per peer", ["peer"]
declarePublicGauge lpt_receiver_distinct_duplicate_messages_count,
declarePublicGauge logos_delivery_lpt_receiver_distinct_duplicate_messages_count,
"number of distinct duplicate messages per peer", ["peer"]
declarePublicGauge lpt_receiver_latencies,
declarePublicGauge logos_delivery_lpt_receiver_latencies,
"Message delivery latency per peer (min-avg-max)", ["peer", "latency"]
declarePublicCounter lpt_receiver_lost_subscription_count,
declarePublicCounter logos_delivery_lpt_receiver_lost_subscription_count,
"number of filter service peer failed PING requests - lost subscription"
declarePublicCounter lpt_publisher_sent_messages_count, "number of messages published"
declarePublicCounter logos_delivery_lpt_publisher_sent_messages_count,
"number of messages published"
declarePublicCounter lpt_publisher_failed_messages_count,
declarePublicCounter logos_delivery_lpt_publisher_failed_messages_count,
"number of messages failed to publish per failure cause", ["cause"]
declarePublicCounter lpt_publisher_sent_bytes, "number of total bytes sent"
declarePublicCounter logos_delivery_lpt_publisher_sent_bytes,
"number of total bytes sent"
declarePublicCounter lpt_service_peer_failure_count,
declarePublicCounter logos_delivery_lpt_service_peer_failure_count,
"number of failure during using service peer [publisher/receiever]", ["role", "agent"]
declarePublicCounter lpt_change_service_peer_count,
declarePublicCounter logos_delivery_lpt_change_service_peer_count,
"number of times [publisher/receiver] had to change service peer", ["role"]
declarePublicGauge lpt_px_peers,
declarePublicGauge logos_delivery_lpt_px_peers,
"Number of peers PeerExchange discovered and can be dialed"
declarePublicGauge lpt_dialed_peers, "Number of peers successfully dialed", ["agent"]
declarePublicGauge logos_delivery_lpt_dialed_peers,
"Number of peers successfully dialed", ["agent"]
declarePublicGauge lpt_dial_failures, "Number of dial failures by cause", ["agent"]
declarePublicGauge logos_delivery_lpt_dial_failures,
"Number of dial failures by cause", ["agent"]
declarePublicHistogram lpt_publish_duration_seconds,
declarePublicHistogram logos_delivery_lpt_publish_duration_seconds,
"duration to lightpush messages",
buckets = [
0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0,

52
apps/liteprotocoltester/monitoring/configuration/dashboards/liter-protocol-test-monitoring.json
View File

@ -99,7 +99,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_px_peers{instance=~\".*publisher.*\"}",
"expr": "(lpt_px_peers{instance=~\".*publisher.*\"} or logos_delivery_lpt_px_peers{instance=~\".*publisher.*\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -204,7 +204,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_dialed_peers{instance=~\".*publisher.*\"}",
"expr": "(lpt_dialed_peers{instance=~\".*publisher.*\"} or logos_delivery_lpt_dialed_peers{instance=~\".*publisher.*\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -221,7 +221,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_dial_failures{instance=~\".*publisher.*\"}",
"expr": "(lpt_dial_failures{instance=~\".*publisher.*\"} or logos_delivery_lpt_dial_failures{instance=~\".*publisher.*\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -301,7 +301,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_px_peers{instance=~\".*receiver.*\"}",
"expr": "(lpt_px_peers{instance=~\".*receiver.*\"} or logos_delivery_lpt_px_peers{instance=~\".*receiver.*\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -406,7 +406,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_dialed_peers{instance=~\".*receivernode.*\"}",
"expr": "(lpt_dialed_peers{instance=~\".*receivernode.*\"} or logos_delivery_lpt_dialed_peers{instance=~\".*receivernode.*\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -423,7 +423,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_dial_failures{instance=~\".*receivernode.*\"}",
"expr": "(lpt_dial_failures{instance=~\".*receivernode.*\"} or logos_delivery_lpt_dial_failures{instance=~\".*receivernode.*\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -540,7 +540,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instace) (lpt_service_peer_failure_count_total{instance=~\".*publishernode.*\", role=\"publisher\"})",
"expr": "sum by(instace) ((lpt_service_peer_failure_count_total{instance=~\".*publishernode.*\", role=\"publisher\"} or logos_delivery_lpt_service_peer_failure_count_total{instance=~\".*publishernode.*\", role=\"publisher\"}))",
"format": "time_series",
"fullMetaSearch": false,
"includeNullMetadata": true,
@ -556,7 +556,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "sum by(instace) (lpt_change_service_peer_count_total{instance=~\".*publishernode.*\", role=\"publisher\"})",
"expr": "sum by(instace) ((lpt_change_service_peer_count_total{instance=~\".*publishernode.*\", role=\"publisher\"} or logos_delivery_lpt_change_service_peer_count_total{instance=~\".*publishernode.*\", role=\"publisher\"}))",
"hide": false,
"instant": false,
"legendFormat": "Peer switch",
@ -659,7 +659,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instace) (lpt_service_peer_failure_count_total{instance=~\".*receivernode.*\", role=\"receiver\"})",
"expr": "sum by(instace) ((lpt_service_peer_failure_count_total{instance=~\".*receivernode.*\", role=\"receiver\"} or logos_delivery_lpt_service_peer_failure_count_total{instance=~\".*receivernode.*\", role=\"receiver\"}))",
"format": "time_series",
"fullMetaSearch": false,
"includeNullMetadata": true,
@ -675,7 +675,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "sum by(instace) (lpt_change_service_peer_count_total{instance=~\".*receivernode.*\", role=\"receiver\"})",
"expr": "sum by(instace) ((lpt_change_service_peer_count_total{instance=~\".*receivernode.*\", role=\"receiver\"} or logos_delivery_lpt_change_service_peer_count_total{instance=~\".*receivernode.*\", role=\"receiver\"}))",
"hide": false,
"instant": false,
"legendFormat": "Peer switch",
@ -688,7 +688,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "sum by(instace) (lpt_receiver_lost_subscription_count_total{instance=~\".*receivernode.*\"})",
"expr": "sum by(instace) ((lpt_receiver_lost_subscription_count_total{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_lost_subscription_count_total{instance=~\".*receivernode.*\"}))",
"hide": false,
"instant": false,
"legendFormat": "Subscription loss - ping fail",
@ -759,7 +759,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "count(\n group(\n last_over_time(lpt_px_peers{instance=~\".*publishernode.*\"}[24h])\n ) by (instance)\n)",
"expr": "count(\n group(\n (last_over_time(lpt_px_peers{instance=~\".*publishernode.*\"}[24h]) or last_over_time(logos_delivery_lpt_px_peers{instance=~\".*publishernode.*\"}[24h]))\n ) by (instance)\n)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -775,7 +775,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "count(\n group(\n last_over_time(lpt_px_peers{instance=~\".*receivernode.*\"}[24h])\n ) by (instance)\n)",
"expr": "count(\n group(\n (last_over_time(lpt_px_peers{instance=~\".*receivernode.*\"}[24h]) or last_over_time(logos_delivery_lpt_px_peers{instance=~\".*receivernode.*\"}[24h]))\n ) by (instance)\n)",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -852,7 +852,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_receiver_sender_peer_count{instance=~\".*receivernode.*\"}",
"expr": "(lpt_receiver_sender_peer_count{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_sender_peer_count{instance=~\".*receivernode.*\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1002,7 +1002,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instace) (lpt_publisher_sent_messages_count_total{instance=~\".*publishernode.*\"})",
"expr": "sum by(instace) ((lpt_publisher_sent_messages_count_total{instance=~\".*publishernode.*\"} or logos_delivery_lpt_publisher_sent_messages_count_total{instance=~\".*publishernode.*\"}))",
"format": "time_series",
"fullMetaSearch": false,
"includeNullMetadata": true,
@ -1020,7 +1020,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instance) (rate(lpt_publisher_sent_messages_count_total{instance=~\".*publishernode.*\"}[$__rate_interval]))",
"expr": "sum by(instance) ((rate(lpt_publisher_sent_messages_count_total{instance=~\".*publishernode.*\"}[$__rate_interval]) or rate(logos_delivery_lpt_publisher_sent_messages_count_total{instance=~\".*publishernode.*\"}[$__rate_interval])))",
"format": "time_series",
"fullMetaSearch": false,
"hide": false,
@ -1157,7 +1157,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sum by(instance) (lpt_receiver_received_messages_count_total{instance=~\".*receivernode.*\"})",
"expr": "sum by(instance) ((lpt_receiver_received_messages_count_total{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_received_messages_count_total{instance=~\".*receivernode.*\"}))",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1173,7 +1173,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sum by(instance) (rate(lpt_receiver_received_messages_count_total{instance=~\".*receivernode.*\"}[$__rate_interval]))",
"expr": "sum by(instance) ((rate(lpt_receiver_received_messages_count_total{instance=~\".*receivernode.*\"}[$__rate_interval]) or rate(logos_delivery_lpt_receiver_received_messages_count_total{instance=~\".*receivernode.*\"}[$__rate_interval])))",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -1318,7 +1318,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instance) (lpt_publisher_sent_bytes_total{instance=~\".*publishernode.*\"})",
"expr": "sum by(instance) ((lpt_publisher_sent_bytes_total{instance=~\".*publishernode.*\"} or logos_delivery_lpt_publisher_sent_bytes_total{instance=~\".*publishernode.*\"}))",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1335,7 +1335,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instance) (rate(lpt_publisher_sent_bytes_total{instance=~\".*publishernode.*\"}[$__rate_interval]))",
"expr": "sum by(instance) ((rate(lpt_publisher_sent_bytes_total{instance=~\".*publishernode.*\"}[$__rate_interval]) or rate(logos_delivery_lpt_publisher_sent_bytes_total{instance=~\".*publishernode.*\"}[$__rate_interval])))",
"format": "time_series",
"fullMetaSearch": false,
"hide": false,
@ -1481,7 +1481,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instance) (lpt_receiver_received_bytes_total{instance=~\".*receivernode.*\"})",
"expr": "sum by(instance) ((lpt_receiver_received_bytes_total{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_received_bytes_total{instance=~\".*receivernode.*\"}))",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1498,7 +1498,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "sum by(instance) (rate(lpt_receiver_received_bytes_total{instance=~\".*receivernode.*\"}[$__rate_interval]))",
"expr": "sum by(instance) ((rate(lpt_receiver_received_bytes_total{instance=~\".*receivernode.*\"}[$__rate_interval]) or rate(logos_delivery_lpt_receiver_received_bytes_total{instance=~\".*receivernode.*\"}[$__rate_interval])))",
"format": "time_series",
"fullMetaSearch": false,
"hide": false,
@ -1639,7 +1639,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "lpt_publisher_failed_messages_count_total{instance=~\".*publishernode.*\"}",
"expr": "(lpt_publisher_failed_messages_count_total{instance=~\".*publishernode.*\"} or logos_delivery_lpt_publisher_failed_messages_count_total{instance=~\".*publishernode.*\"})",
"format": "time_series",
"fullMetaSearch": false,
"includeNullMetadata": true,
@ -1745,7 +1745,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_receiver_duplicate_messages_count_total{instance=~\".*receivernode.*\"}",
"expr": "(lpt_receiver_duplicate_messages_count_total{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_duplicate_messages_count_total{instance=~\".*receivernode.*\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1761,7 +1761,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_receiver_distinct_duplicate_messages_count{instance=~\".*receivernode.*\"}",
"expr": "(lpt_receiver_distinct_duplicate_messages_count{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_distinct_duplicate_messages_count{instance=~\".*receivernode.*\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -1867,7 +1867,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "lpt_receiver_missing_messages_count{instance=~\".*receivernode.*\"}",
"expr": "(lpt_receiver_missing_messages_count{instance=~\".*receivernode.*\"} or logos_delivery_lpt_receiver_missing_messages_count{instance=~\".*receivernode.*\"})",
"format": "time_series",
"fullMetaSearch": false,
"hide": false,

16
apps/liteprotocoltester/publisher.nim
View File

@ -155,7 +155,9 @@ proc publishMessages(
let msgHash = computeMessageHash(lightpushPubsubTopic, message).to0xHex
if wlpRes.isOk():
lpt_publish_duration_seconds.observe(publishDuration.milliseconds.float / 1000)
logos_delivery_lpt_publish_duration_seconds.observe(
publishDuration.milliseconds.float / 1000
)
sentMessages[messagesSent] = (hash: msgHash, relayed: true)
notice "published message using lightpush",
@ -165,8 +167,8 @@ proc publishMessages(
pubsubTopic = lightpushPubsubTopic,
hash = msgHash
inc(messagesSent)
lpt_publisher_sent_messages_count.inc()
lpt_publisher_sent_bytes.inc(amount = msgSize.int64)
logos_delivery_lpt_publisher_sent_messages_count.inc()
logos_delivery_lpt_publisher_sent_bytes.inc(amount = msgSize.int64)
if noFailedPush > 0:
noFailedPush -= 1
else:
@ -175,14 +177,16 @@ proc publishMessages(
error "failed to publish message using lightpush",
err = wlpRes.error, hash = msgHash
inc(failedToSendCount)
lpt_publisher_failed_messages_count.inc(labelValues = [wlpRes.error])
logos_delivery_lpt_publisher_failed_messages_count.inc(
labelValues = [wlpRes.error]
)
if not wlpRes.error.toLower().contains("dial"):
# retry sending after shorter wait
await sleepAsync(2.seconds)
continue
else:
noFailedPush += 1
lpt_service_peer_failure_count.inc(
logos_delivery_lpt_service_peer_failure_count.inc(
labelValues = ["publisher", actualServicePeer.getAgent()]
)
if not preventPeerSwitch and noFailedPush > maxFailedPush:
@ -200,7 +204,7 @@ proc publishMessages(
noFailedPush = 0
noOfServicePeerSwitches += 1
lpt_change_service_peer_count.inc(labelValues = ["publisher"])
logos_delivery_lpt_change_service_peer_count.inc(labelValues = ["publisher"])
continue # try again with new peer without delay
await sleepAsync(messageInterval)

6
apps/liteprotocoltester/receiver.nim
View File

@ -65,7 +65,7 @@ proc maintainSubscription(
if isFirstPingOnNewPeer == false:
# Very first ping expected to fail as we have not yet subscribed at all
lpt_receiver_lost_subscription_count.inc()
logos_delivery_lpt_receiver_lost_subscription_count.inc()
isFirstPingOnNewPeer = false
# No subscription found. Let's subscribe.
error "ping failed.", error = pingErr
@ -83,7 +83,7 @@ proc maintainSubscription(
continue
noFailedSubscribes += 1
lpt_service_peer_failure_count.inc(
logos_delivery_lpt_service_peer_failure_count.inc(
labelValues = ["receiver", actualFilterPeer.getAgent()]
)
error "Subscribe request failed.",
@ -108,7 +108,7 @@ proc maintainSubscription(
codec = filterPubsubTopic, peer = constructMultiaddrStr(actualFilterPeer)
noFailedSubscribes = 0
lpt_change_service_peer_count.inc(labelValues = ["receiver"])
logos_delivery_lpt_change_service_peer_count.inc(labelValues = ["receiver"])
isFirstPingOnNewPeer = true
else:
await sleepAsync(SubscriptionMaintenanceMs)

8
apps/liteprotocoltester/service_peer_management.nim
View File

@ -104,7 +104,7 @@ proc tryCallAllPxPeers*(
var supportivePeers = pm.switch.peerStore.getPeersByCapability(capability)
lpt_px_peers.set(supportivePeers.len)
logos_delivery_lpt_px_peers.set(supportivePeers.len)
info "Found supportive peers count", count = supportivePeers.len()
info "Found supportive peers", supportivePeers = $supportivePeers
if supportivePeers.len == 0:
@ -129,15 +129,15 @@ proc tryCallAllPxPeers*(
peer = constructMultiaddrStr(randomPeer),
agent = randomPeer.getAgent(),
codec = codec
lpt_dialed_peers.inc(labelValues = [randomPeer.getAgent()])
logos_delivery_lpt_dialed_peers.inc(labelValues = [randomPeer.getAgent()])
else:
lpt_dial_failures.inc(labelValues = [randomPeer.getAgent()])
logos_delivery_lpt_dial_failures.inc(labelValues = [randomPeer.getAgent()])
error "Dialing failed",
peer = constructMultiaddrStr(randomPeer),
agent = randomPeer.getAgent(),
codec = codec
else:
lpt_dial_failures.inc(labelValues = [randomPeer.getAgent()])
logos_delivery_lpt_dial_failures.inc(labelValues = [randomPeer.getAgent()])
error "Timeout dialing service peer",
peer = constructMultiaddrStr(randomPeer),
agent = randomPeer.getAgent(),

26
apps/liteprotocoltester/statistics.nim
View File

@ -75,7 +75,9 @@ proc addMessage*(
prevIndex: self.helper.prevIndex,
),
)
lpt_receiver_received_bytes.inc(labelValues = [sender], amount = msg.size.int64)
logos_delivery_lpt_receiver_received_bytes.inc(
labelValues = [sender], amount = msg.size.int64
)
if self.received.hasKeyOrPut(msg.index, currentArrived):
inc(self.duplicateCount)
self.helper.duplicates.mgetOrPut(msg.index, (msgHash, 0, msg.size)).dupCount.inc()
@ -83,8 +85,8 @@ proc addMessage*(
index = msg.index,
hash = msgHash,
times_duplicated = self.helper.duplicates[msg.index].dupCount
lpt_receiver_duplicate_messages_count.inc(labelValues = [sender])
lpt_receiver_distinct_duplicate_messages_count.set(
logos_delivery_lpt_receiver_duplicate_messages_count.inc(labelValues = [sender])
logos_delivery_lpt_receiver_distinct_duplicate_messages_count.set(
labelValues = [sender], value = self.helper.duplicates.len()
)
return
@ -102,8 +104,8 @@ proc addMessage*(
self.helper.prevIndex = msg.index
self.helper.prevArrivedAt = currentArrived.info.arrivedAt
inc(self.receivedMessages)
lpt_receiver_received_messages_count.inc(labelValues = [sender])
lpt_receiver_missing_messages_count.set(
logos_delivery_lpt_receiver_received_messages_count.inc(labelValues = [sender])
logos_delivery_lpt_receiver_missing_messages_count.set(
labelValues = [sender], value = (self.helper.maxIndex - self.receivedMessages).int64
)
@ -121,7 +123,7 @@ proc addMessage*(
discard catch:
self[peerId].addMessage(shortSenderId, msg, msgHash)
lpt_receiver_sender_peer_count.set(value = self.len)
logos_delivery_lpt_receiver_sender_peer_count.set(value = self.len)
proc lastMessageArrivedAt*(self: Statistics): Opt[Moment] =
if self.receivedMessages > 0:
@ -193,9 +195,15 @@ proc dupMsgs(self: Statistics): string =
proc echoStat*(self: Statistics, peerId: string) =
let (minL, maxL, avgL) = self.calcLatency()
lpt_receiver_latencies.set(labelValues = [peerId, "min"], value = minL.nanos())
lpt_receiver_latencies.set(labelValues = [peerId, "avg"], value = avgL.nanos())
lpt_receiver_latencies.set(labelValues = [peerId, "max"], value = maxL.nanos())
logos_delivery_lpt_receiver_latencies.set(
labelValues = [peerId, "min"], value = minL.nanos()
)
logos_delivery_lpt_receiver_latencies.set(
labelValues = [peerId, "avg"], value = avgL.nanos()
)
logos_delivery_lpt_receiver_latencies.set(
labelValues = [peerId, "max"], value = maxL.nanos()
)
let printable = catch:
"""*------------------------------------------------------------------------------------------*

25
apps/networkmonitor/networkmonitor.nim
View File

@ -50,7 +50,7 @@ proc setDiscoveredPeersCapabilities(routingTableNodes: seq[waku_enr.Record]) =
routingTableNodes.countIt(it.supportsCapability(capability))
info "capabilities as per ENR waku flag",
capability = capability, amount = nOfNodesWithCapability
networkmonitor_peer_type_as_per_enr.set(
logos_delivery_networkmonitor_peer_type_as_per_enr.set(
int64(nOfNodesWithCapability), labelValues = [$capability]
)
@ -69,7 +69,9 @@ proc setDiscoveredPeersCluster(routingTableNodes: seq[Node]) =
clusters.inc(relayShard.clusterId)
for (key, value) in clusters.pairs:
networkmonitor_peer_cluster_as_per_enr.set(int64(value), labelValues = [$key])
logos_delivery_networkmonitor_peer_cluster_as_per_enr.set(
int64(value), labelValues = [$key]
)
proc analyzePeer(
customPeerInfo: CustomPeerInfoRef,
@ -101,7 +103,7 @@ proc analyzePeer(
customPeerInfo.connError = ""
info "successfully pinged peer", peer = peerInfo, duration = pingDelay.millis
networkmonitor_peer_ping.observe(pingDelay.millis)
logos_delivery_networkmonitor_peer_ping.observe(pingDelay.millis)
# We are using a smoothed moving average
customPeerInfo.avgPingDuration =
@ -252,12 +254,15 @@ proc updateMetrics(allPeersRef: CustomPeersTableRef) {.gcsafe.} =
else:
failedPeers += 1
networkmonitor_peer_count.set(int64(connectedPeers), labelValues = ["true"])
networkmonitor_peer_count.set(int64(failedPeers), labelValues = ["false"])
# update count on each protocol
logos_delivery_networkmonitor_peer_count.set(
int64(connectedPeers), labelValues = ["true"]
)
logos_delivery_networkmonitor_peer_count.set(
int64(failedPeers), labelValues = ["false"]
) # update count on each protocol
for protocol in allProtocols.keys():
let countOfProtocols = allProtocols.mgetOrPut(protocol, 0)
networkmonitor_peer_type_as_per_protocol.set(
logos_delivery_networkmonitor_peer_type_as_per_protocol.set(
int64(countOfProtocols), labelValues = [protocol]
)
info "supported protocols in the network",
@ -266,7 +271,7 @@ proc updateMetrics(allPeersRef: CustomPeersTableRef) {.gcsafe.} =
# update count on each user-agent
for userAgent in allAgentStrings.keys():
let countOfUserAgent = allAgentStrings.mgetOrPut(userAgent, 0)
networkmonitor_peer_user_agents.set(
logos_delivery_networkmonitor_peer_user_agents.set(
int64(countOfUserAgent), labelValues = [userAgent]
)
info "user agents participating in the network",
@ -274,7 +279,9 @@ proc updateMetrics(allPeersRef: CustomPeersTableRef) {.gcsafe.} =
for country in countries.keys():
let peerCount = countries.mgetOrPut(country, 0)
networkmonitor_peer_country_count.set(int64(peerCount), labelValues = [country])
logos_delivery_networkmonitor_peer_country_count.set(
int64(peerCount), labelValues = [country]
)
info "number of peers per country", country = country, count = peerCount
proc populateInfoFromIp(

14
apps/networkmonitor/networkmonitor_metrics.nim
View File

@ -21,28 +21,28 @@ logScope:
#discovery_message_requests_outgoing_total{response=""}
#discovery_message_requests_outgoing_total{response="no_response"}
declarePublicGauge networkmonitor_peer_type_as_per_enr,
declarePublicGauge logos_delivery_networkmonitor_peer_type_as_per_enr,
"Number of peers supporting each capability according to the ENR",
labels = ["capability"]
declarePublicGauge networkmonitor_peer_cluster_as_per_enr,
declarePublicGauge logos_delivery_networkmonitor_peer_cluster_as_per_enr,
"Number of peers on each cluster according to the ENR", labels = ["cluster"]
declarePublicGauge networkmonitor_peer_type_as_per_protocol,
declarePublicGauge logos_delivery_networkmonitor_peer_type_as_per_protocol,
"Number of peers supporting each protocol, after a successful connection) ",
labels = ["protocols"]
declarePublicGauge networkmonitor_peer_user_agents,
declarePublicGauge logos_delivery_networkmonitor_peer_user_agents,
"Number of peers with each user agent", labels = ["user_agent"]
declarePublicHistogram networkmonitor_peer_ping,
declarePublicHistogram logos_delivery_networkmonitor_peer_ping,
"Histogram tracking ping durations for discovered peers",
buckets = [10.0, 20.0, 50.0, 100.0, 200.0, 300.0, 500.0, 800.0, 1000.0, 2000.0, Inf]
declarePublicGauge networkmonitor_peer_count,
declarePublicGauge logos_delivery_networkmonitor_peer_count,
"Number of discovered peers", labels = ["connected"]
declarePublicGauge networkmonitor_peer_country_count,
declarePublicGauge logos_delivery_networkmonitor_peer_country_count,
"Number of peers per country", labels = ["country"]
type

38
apps/sonda/monitoring/configuration/dashboards/nwaku-monitoring.json
View File

@ -97,7 +97,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "rate(waku_histogram_message_size_bucket[1h])/scalar(rate(waku_histogram_message_size_count[1h]))*100",
"expr": "(rate(waku_histogram_message_size_bucket[1h]) or rate(logos_delivery_histogram_message_size_bucket[1h]))/scalar((rate(waku_histogram_message_size_count[1h]) or rate(logos_delivery_histogram_message_size_count[1h])))*100",
"format": "heatmap",
"instant": true,
"legendFormat": "__auto",
@ -166,7 +166,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_histogram_message_size_sum[1h])/rate(waku_histogram_message_size_count[1h])",
"expr": "(rate(waku_histogram_message_size_sum[1h]) or rate(logos_delivery_histogram_message_size_sum[1h]))/(rate(waku_histogram_message_size_count[1h]) or rate(logos_delivery_histogram_message_size_count[1h]))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -229,7 +229,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "histogram_quantile(0.75, rate(waku_histogram_message_size_bucket[1h]))",
"expr": "histogram_quantile(0.75, (rate(waku_histogram_message_size_bucket[1h]) or rate(logos_delivery_histogram_message_size_bucket[1h])))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -296,7 +296,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, rate(waku_histogram_message_size_bucket[1h]))",
"expr": "histogram_quantile(0.99, (rate(waku_histogram_message_size_bucket[1h]) or rate(logos_delivery_histogram_message_size_bucket[1h])))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -390,7 +390,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "(increase(waku_node_messages_total[1m]))/60",
"expr": "((increase(waku_node_messages_total[1m]) or increase(logos_delivery_node_messages_total[1m])))/60",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -483,7 +483,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_histogram_message_size_sum/waku_histogram_message_size_count",
"expr": "(waku_histogram_message_size_sum or logos_delivery_histogram_message_size_sum)/(waku_histogram_message_size_count or logos_delivery_histogram_message_size_count)",
"format": "heatmap",
"instant": false,
"legendFormat": "{{instance}}",
@ -556,7 +556,7 @@
},
"editorMode": "builder",
"exemplar": false,
"expr": "waku_version{instance=\"nwaku:8003\"}",
"expr": "(waku_version{instance=\"nwaku:8003\"} or logos_delivery_version{instance=\"nwaku:8003\"})",
"format": "table",
"instant": true,
"legendFormat": "__auto",
@ -1054,7 +1054,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "waku_connected_peers",
"expr": "(waku_connected_peers or logos_delivery_connected_peers)",
"legendFormat": "{{direction}}_{{protocol}}",
"range": true,
"refId": "A"
@ -1117,7 +1117,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "waku_peer_store_size",
"expr": "(waku_peer_store_size or logos_delivery_peer_store_size)",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -1688,7 +1688,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_number_registered_memberships",
"expr": "(waku_rln_number_registered_memberships or logos_delivery_rln_number_registered_memberships)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1787,7 +1787,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_proof_generation_duration_seconds",
"expr": "(waku_rln_proof_generation_duration_seconds or logos_delivery_rln_proof_generation_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1886,7 +1886,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_proof_verification_duration_seconds",
"expr": "(waku_rln_proof_verification_duration_seconds or logos_delivery_rln_proof_verification_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -1985,7 +1985,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_membership_insertion_duration_seconds",
"expr": "(waku_rln_membership_insertion_duration_seconds or logos_delivery_rln_membership_insertion_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -2084,7 +2084,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_membership_credentials_import_duration_seconds",
"expr": "(waku_rln_membership_credentials_import_duration_seconds or logos_delivery_rln_membership_credentials_import_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -2183,7 +2183,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_messages_total_total",
"expr": "(waku_rln_messages_total_total or logos_delivery_rln_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -2282,7 +2282,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_proof_verification_total_total",
"expr": "(waku_rln_proof_verification_total_total or logos_delivery_rln_proof_verification_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -2381,7 +2381,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_invalid_messages_total_total",
"expr": "(waku_rln_invalid_messages_total_total or logos_delivery_rln_invalid_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -2480,7 +2480,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_spam_messages_total_total",
"expr": "(waku_rln_spam_messages_total_total or logos_delivery_rln_spam_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -2579,7 +2579,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_invalid_messages_total_total",
"expr": "(waku_rln_invalid_messages_total_total or logos_delivery_rln_invalid_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,

42
docs/operators/how-to/monitor.md
View File

@ -106,6 +106,44 @@ which you can [import to your Grafana instance](https://grafana.com/docs/grafana
Some of the most important metrics to keep an eye on include:
- `libp2p_peers` as an indication of how many peers your node is connected to,
- `waku_node_messages_total` to view the total amount of network traffic relayed by your node and
- `waku_node_errors` as a rough indication of basic operating errors logged by the node.
- `logos_delivery_node_messages_total` to view the total amount of network traffic relayed by your node and
- `logos_delivery_node_errors` as a rough indication of basic operating errors logged by the node.
## Metric names
Every metric exported by the node is prefixed with `logos_delivery_`.
Metrics coming from libraries the node builds on (`libp2p_*`, `nim_gc_*`, `process_*`)
keep their own names and are unaffected.
Nodes before this rename exported the same metrics under a `waku_` prefix (and a few
under no prefix at all, such as `query_count` or `event_loop_load`). The dashboards in
[`metrics/`](https://github.com/logos-messaging/logos-delivery/tree/master/metrics) query
both spellings, so they keep working across the upgrade and over historical data:
```
(waku_store_queries_total or logos_delivery_store_queries_total)
```
Place the `or` around the leaf, inside any aggregation — not around the whole
expression. `or` keeps its right operand only for label sets that are absent from the
left, so this undercounts while a fleet is mid-upgrade:
```
sum by (type)(waku_node_errors_total) or sum by (type)(logos_delivery_node_errors_total)
```
Both sides collapse to the same label set, so the right one is discarded entirely and
every already-upgraded node vanishes from the panel. Write this instead:
```
sum by (type)(waku_node_errors_total or logos_delivery_node_errors_total)
```
When the metric sits inside a range vector, duplicate the whole call —
`(rate(old[5m]) or rate(new[5m]))` — since `(old or new)[5m]` is not valid PromQL.
If you maintain your own dashboards, alert rules or recording rules, apply the same
change before upgrading. Once every node you scrape runs a release with the new names
and the old samples have aged out of your Prometheus retention window, the `waku_`
alternative can be dropped.

6
examples/lightpush_mix/lightpush_publisher_mix.nim
View File

@ -176,12 +176,12 @@ proc setupAndPublish(rng: crypto.Rng, conf: LightPushMixConf) {.async.} =
)
).isOkOr:
error "failed to publish message via mix", error = error.desc
lp_mix_failed.inc(labelValues = ["publish_error"])
logos_delivery_lp_mix_failed.inc(labelValues = ["publish_error"])
return
let latency = float64(getNowInNanosecondTime() - startTime) / 1_000_000.0
lp_mix_latency.observe(latency)
lp_mix_success.inc()
logos_delivery_lp_mix_latency.observe(latency)
logos_delivery_lp_mix_success.inc()
notice "published message",
text = text,
timestamp = message.timestamp,

7
examples/lightpush_mix/lightpush_publisher_mix_metrics.nim
View File

@ -2,10 +2,11 @@
import metrics
declarePublicCounter lp_mix_success, "number of lightpush messages sent via mix"
declarePublicCounter logos_delivery_lp_mix_success,
"number of lightpush messages sent via mix"
declarePublicCounter lp_mix_failed,
declarePublicCounter logos_delivery_lp_mix_failed,
"number of lightpush messages failed via mix", labels = ["error"]
declarePublicHistogram lp_mix_latency,
declarePublicHistogram logos_delivery_lp_mix_latency,
"lightpush publish latency via mix in milliseconds"

12
logos_delivery/waku/common/databases/db_postgres/dbconn.nim
View File

@ -265,7 +265,7 @@ proc dbConnQuery*(
return err("error in dbConnQuery calling sendQuery: " & $error)
let sendDuration = getTime().toUnixFloat() - queryStartTime
query_time_secs.set(sendDuration, [metricLabel, "sendToDBQuery"])
logos_delivery_query_time_secs.set(sendDuration, [metricLabel, "sendToDBQuery"])
queryStartTime = getTime().toUnixFloat()
@ -273,9 +273,9 @@ proc dbConnQuery*(
return err("error in dbConnQuery calling waitQueryToFinish: " & $error)
let waitDuration = getTime().toUnixFloat() - queryStartTime
query_time_secs.set(waitDuration, [metricLabel, "waitFinish"])
logos_delivery_query_time_secs.set(waitDuration, [metricLabel, "waitFinish"])
query_count.inc(labelValues = [metricLabel])
logos_delivery_query_count.inc(labelValues = [metricLabel])
if "insert" notin ($query).toLower():
info "dbConnQuery",
@ -306,7 +306,7 @@ proc dbConnQueryPrepared*(
return err("error in dbConnQueryPrepared calling sendQuery: " & $error)
let sendDuration = getTime().toUnixFloat() - queryStartTime
query_time_secs.set(sendDuration, [stmtName, "sendToDBQuery"])
logos_delivery_query_time_secs.set(sendDuration, [stmtName, "sendToDBQuery"])
queryStartTime = getTime().toUnixFloat()
@ -314,9 +314,9 @@ proc dbConnQueryPrepared*(
return err("error in dbConnQueryPrepared calling waitQueryToFinish: " & $error)
let waitDuration = getTime().toUnixFloat() - queryStartTime
query_time_secs.set(waitDuration, [stmtName, "waitFinish"])
logos_delivery_query_time_secs.set(waitDuration, [stmtName, "waitFinish"])
query_count.inc(labelValues = [stmtName])
logos_delivery_query_count.inc(labelValues = [stmtName])
if "insert" notin stmtName.toLower():
info "dbConnQueryPrepared",

4
logos_delivery/waku/common/databases/db_postgres/query_metrics.nim
View File

@ -1,9 +1,9 @@
import metrics, tables
declarePublicGauge query_time_secs,
declarePublicGauge logos_delivery_query_time_secs,
"query time measured in nanoseconds", labels = ["query", "phase"]
declarePublicCounter query_count,
declarePublicCounter logos_delivery_query_count,
"number of times a query is being performed", labels = ["query"]
## Maps parts of the possible known queries with a fixed and shorter query label.

6
logos_delivery/waku/common/rate_limit/request_limiter.nim
View File

@ -119,16 +119,16 @@ template checkUsageLimit*(
) =
if t.checkUsage(proto, conn):
let requestStartTime = Moment.now()
waku_service_requests.inc(labelValues = [proto, "served"])
logos_delivery_service_requests.inc(labelValues = [proto, "served"])
bodyWithinLimit
let requestDuration = Moment.now() - requestStartTime
waku_service_request_handling_duration_seconds.observe(
logos_delivery_service_request_handling_duration_seconds.observe(
requestDuration.milliseconds.float / 1000, labelValues = [proto]
)
else:
waku_service_requests.inc(labelValues = [proto, "rejected"])
logos_delivery_service_requests.inc(labelValues = [proto, "rejected"])
bodyRejected
# TODO: review these ratio assumptions! Debatable!

10
logos_delivery/waku/common/rate_limit/service_metrics.nim
View File

@ -4,22 +4,22 @@ import results, metrics, setting
export metrics
declarePublicGauge waku_service_requests_limit,
declarePublicGauge logos_delivery_service_requests_limit,
"Applied rate limit of non-relay service", ["service"]
declarePublicCounter waku_service_requests,
declarePublicCounter logos_delivery_service_requests,
"number of non-relay service requests received", ["service", "state"]
declarePublicCounter waku_service_network_bytes,
declarePublicCounter logos_delivery_service_network_bytes,
"total incoming traffic of specific waku services", labels = ["service", "direction"]
proc setServiceLimitMetric*(service: string, limit: Opt[RateLimitSetting]) =
if limit.isSome() and not limit.get().isUnlimited():
waku_service_requests_limit.set(
logos_delivery_service_requests_limit.set(
limit.get().calculateLimitPerSecond(), labelValues = [service]
)
declarePublicHistogram waku_service_request_handling_duration_seconds,
declarePublicHistogram logos_delivery_service_request_handling_duration_seconds,
"duration of non-relay service handling",
labels = ["service"],
buckets = [

6
logos_delivery/waku/common/rate_limit/single_token_limiter.nim
View File

@ -56,14 +56,14 @@ template checkUsageLimit*(
) =
if t.checkUsage(proto):
let requestStartTime = Moment.now()
waku_service_requests.inc(labelValues = [proto, "served"])
logos_delivery_service_requests.inc(labelValues = [proto, "served"])
bodyWithinLimit
let requestDuration = Moment.now() - requestStartTime
waku_service_request_handling_duration_seconds.observe(
logos_delivery_service_request_handling_duration_seconds.observe(
requestDuration.milliseconds.float / 1000, labelValues = [proto]
)
else:
waku_service_requests.inc(labelValues = [proto, "rejected"])
logos_delivery_service_requests.inc(labelValues = [proto, "rejected"])
bodyRejected

13
logos_delivery/waku/discovery/waku_discv5.nim
View File

@ -18,9 +18,10 @@ import
export protocol, waku_enr
declarePublicGauge waku_discv5_discovered_per_shard,
declarePublicGauge logos_delivery_discv5_discovered_per_shard,
"number of nodes discovered by each shard", labels = ["shard"]
declarePublicCounter waku_discv5_errors, "number of waku discv5 errors", ["type"]
declarePublicCounter logos_delivery_discv5_errors,
"number of waku discv5 errors", ["type"]
logScope:
topics = "waku discv5"
@ -240,16 +241,16 @@ proc findRandomPeers*(
for record in discoveredRecords:
let typedRecord = record.toTyped().valueOr:
# If we can't parse the record, skip it
waku_discv5_errors.inc(labelValues = ["ParseFailure"])
logos_delivery_discv5_errors.inc(labelValues = ["ParseFailure"])
continue
let relayShards = typedRecord.relaySharding().valueOr:
# If no relay sharding info, skip it
waku_discv5_errors.inc(labelValues = ["NoShardInfo"])
logos_delivery_discv5_errors.inc(labelValues = ["NoShardInfo"])
continue
for shardId in relayShards.shardIds:
waku_discv5_discovered_per_shard.inc(labelValues = [$shardId])
logos_delivery_discv5_discovered_per_shard.inc(labelValues = [$shardId])
return discoveredRecords
@ -273,7 +274,7 @@ proc searchLoop(wd: WakuDiscoveryV5) {.async.} =
let peerInfo = record.toRemotePeerInfo().valueOr:
## in case of error, we keep track of it for debugging purposes
wrongRecordsReasons.add(($record, $error))
waku_discv5_errors.inc(labelValues = [$error])
logos_delivery_discv5_errors.inc(labelValues = [$error])
continue
discoveredPeers.add(peerInfo)

11
logos_delivery/waku/discovery/waku_dnsdisc.nim
View File

@ -22,8 +22,9 @@ import ../waku_core
export client
declarePublicGauge waku_dnsdisc_discovered, "number of nodes discovered"
declarePublicCounter waku_dnsdisc_errors, "number of waku dnsdisc errors", ["type"]
declarePublicGauge logos_delivery_dnsdisc_discovered, "number of nodes discovered"
declarePublicCounter logos_delivery_dnsdisc_errors,
"number of waku dnsdisc errors", ["type"]
logScope:
topics = "waku dnsdisc"
@ -53,7 +54,7 @@ proc findPeers*(
tree = (await syncTree(wdd.resolver, wdd.client.loc)).tryGet()
except Exception:
error "Failed to synchronise client tree"
waku_dnsdisc_errors.inc(labelValues = ["tree_sync_failure"])
logos_delivery_dnsdisc_errors.inc(labelValues = ["tree_sync_failure"])
return err("Node discovery failed")
let discoveredEnr = tree.getNodes().mapIt(it.record)
@ -69,12 +70,12 @@ proc findPeers*(
# Convert discovered ENR to RemotePeerInfo and add to discovered nodes
let peerInfo = enr.toRemotePeerInfo().valueOr:
error "Failed to convert ENR to peer info", enr = $enr, error = error
waku_dnsdisc_errors.inc(labelValues = ["peer_info_failure"])
logos_delivery_dnsdisc_errors.inc(labelValues = ["peer_info_failure"])
continue
discoveredNodes.add(peerInfo)
if discoveredNodes.len > 0:
info "Successfully discovered nodes", count = discoveredNodes.len
waku_dnsdisc_discovered.inc(discoveredNodes.len.int64)
logos_delivery_dnsdisc_discovered.inc(discoveredNodes.len.int64)
return ok(discoveredNodes)

4
logos_delivery/waku/factory/validator_signed.nim
View File

@ -15,7 +15,7 @@ const MessageWindowInSec = 5 * 60 # +- 5 minutes
import ./waku_conf, ../waku_relay/protocol, ../waku_core
declarePublicCounter waku_msg_validator_signed_outcome,
declarePublicCounter logos_delivery_msg_validator_signed_outcome,
"number of messages for each validation outcome", ["result"]
# Application level message hash
@ -74,7 +74,7 @@ proc addSignedShardsValidator*(
if outcome != errors.ValidationResult.Accept:
info "signed topic validation failed",
topic = topic, publicShardKey = protectedShard.key
waku_msg_validator_signed_outcome.inc(labelValues = [$outcome])
logos_delivery_msg_validator_signed_outcome.inc(labelValues = [$outcome])
return outcome
return errors.ValidationResult.Accept

14
logos_delivery/waku/node/health_monitor/event_loop_monitor.nim
View File

@ -6,11 +6,11 @@ import chronos, chronicles, metrics
logScope:
topics = "waku event_loop_monitor"
declarePublicGauge event_loop_load,
declarePublicGauge logos_delivery_event_loop_load,
"chronos event loop load EWMA by window (1.0 = sustained lag at MaxAcceptedLag)",
labels = ["window"]
declarePublicCounter event_loop_accumulated_lag_secs,
declarePublicCounter logos_delivery_event_loop_accumulated_lag_secs,
"chronos event loop total accumulated lag in seconds since node start"
type OnLagChange* = proc(lagTooHigh: bool) {.gcsafe, raises: [].}
@ -21,7 +21,7 @@ proc eventLoopMonitorLoop*(onLagChange: OnLagChange = nil) {.async.} =
##
## The lag is normalised against `MaxAcceptedLag` and tracked as an EWMA
## over 1, 5, and 15-minute windows (Unix load-average decay model),
## exposed via the `event_loop_load` gauge (labelled by window: 1m/5m/15m):
## exposed via the `logos_delivery_event_loop_load` gauge (labelled by window: 1m/5m/15m):
##
## load < 1.0 → within budget
## load = 1.0 → sustained lag at MaxAcceptedLag (fully loaded)
@ -58,15 +58,15 @@ proc eventLoopMonitorLoop*(onLagChange: OnLagChange = nil) {.async.} =
let lagSecs = lag.nanoseconds.float64 / 1_000_000_000.0
let load = lagSecs / maxAcceptedLagSecs
event_loop_accumulated_lag_secs.inc(lagSecs)
logos_delivery_event_loop_accumulated_lag_secs.inc(lagSecs)
ewma1m = alpha1m * load + (1.0 - alpha1m) * ewma1m
ewma5m = alpha5m * load + (1.0 - alpha5m) * ewma5m
ewma15m = alpha15m * load + (1.0 - alpha15m) * ewma15m
event_loop_load.set(round(ewma1m, 4), labelValues = ["1m"])
event_loop_load.set(round(ewma5m, 4), labelValues = ["5m"])
event_loop_load.set(round(ewma15m, 4), labelValues = ["15m"])
logos_delivery_event_loop_load.set(round(ewma1m, 4), labelValues = ["1m"])
logos_delivery_event_loop_load.set(round(ewma5m, 4), labelValues = ["5m"])
logos_delivery_event_loop_load.set(round(ewma15m, 4), labelValues = ["15m"])
let lagIsHigh = lag > MaxAcceptedLag

2
logos_delivery/waku/node/health_monitor/node_health_monitor.nim
View File

@ -624,7 +624,7 @@ proc keepAliveLoop(
let successfulPings = await parallelPings(node, peersToPing)
if successfulPings != numPeersToPing:
waku_node_errors.inc(
logos_delivery_node_errors.inc(
amount = numPeersToPing - successfulPings, labelValues = ["keep_alive_failure"]
)

17
logos_delivery/waku/node/node_telemetry.nim
View File

@ -2,23 +2,24 @@
import metrics
declarePublicGauge waku_version,
declarePublicGauge logos_delivery_version,
"Waku version info (in git describe format)", ["version"]
declarePublicCounter waku_node_errors, "number of wakunode errors", ["type"]
declarePublicCounter logos_delivery_node_errors, "number of wakunode errors", ["type"]
declarePublicGauge waku_lightpush_peers, "number of lightpush peers"
declarePublicGauge logos_delivery_lightpush_peers, "number of lightpush peers"
declarePublicGauge waku_filter_peers, "number of filter peers"
declarePublicGauge logos_delivery_filter_peers, "number of filter peers"
declarePublicGauge waku_store_peers, "number of store peers"
declarePublicGauge logos_delivery_store_peers, "number of store peers"
declarePublicGauge waku_px_peers,
declarePublicGauge logos_delivery_px_peers,
"number of peers (in the node's peerManager) supporting the peer exchange protocol"
declarePublicCounter waku_node_messages, "number of messages received", ["type"]
declarePublicCounter logos_delivery_node_messages,
"number of messages received", ["type"]
declarePublicHistogram waku_histogram_message_size,
declarePublicHistogram logos_delivery_histogram_message_size,
"message size histogram in kB",
buckets = [
0.0, 1.0, 3.0, 5.0, 15.0, 50.0, 75.0, 100.0, 125.0, 150.0, 500.0, 700.0, 1000.0, Inf

54
logos_delivery/waku/node/peer_manager/peer_manager.nim
View File

@ -31,24 +31,26 @@ import
export waku_peer_store, peer_storage, peers
declareCounter waku_peers_dials, "Number of peer dials", ["outcome"]
declareCounter logos_delivery_peers_dials, "Number of peer dials", ["outcome"]
# TODO: Populate from PeerStore.Source when ready
declarePublicCounter waku_node_conns_initiated,
declarePublicCounter logos_delivery_node_conns_initiated,
"Number of connections initiated", ["source"]
declarePublicCounter waku_peers_errors, "Number of peer manager errors", ["type"]
declarePublicGauge waku_connected_peers,
declarePublicCounter logos_delivery_peers_errors,
"Number of peer manager errors", ["type"]
declarePublicGauge logos_delivery_connected_peers,
"Number of physical connections per direction and protocol",
labels = ["direction", "protocol"]
declarePublicGauge waku_connected_peers_per_shard,
declarePublicGauge logos_delivery_connected_peers_per_shard,
"Number of physical connections per shard", labels = ["shard"]
declarePublicGauge waku_connected_peers_per_agent,
declarePublicGauge logos_delivery_connected_peers_per_agent,
"Number of physical connections per agent", labels = ["agent"]
declarePublicGauge waku_streams_peers,
declarePublicGauge logos_delivery_streams_peers,
"Number of streams per direction and protocol", labels = ["direction", "protocol"]
declarePublicGauge waku_peer_store_size, "Number of peers managed by the peer store"
declarePublicGauge waku_service_peers,
declarePublicGauge logos_delivery_peer_store_size,
"Number of peers managed by the peer store"
declarePublicGauge logos_delivery_service_peers,
"Service peer protocol and multiaddress ", labels = ["protocol", "peerId"]
declarePublicGauge waku_total_unique_peers, "total number of unique peers"
declarePublicGauge logos_delivery_total_unique_peers, "total number of unique peers"
logScope:
topics = "waku node peer_manager"
@ -140,7 +142,7 @@ proc insertOrReplace(ps: PeerStorage, remotePeerInfo: RemotePeerInfo) {.gcsafe.}
## Insert peer entry into persistent storage, or replace existing entry with updated info
ps.put(remotePeerInfo).isOkOr:
warn "failed to store peers", err = error
waku_peers_errors.inc(labelValues = ["storage_failure"])
logos_delivery_peers_errors.inc(labelValues = ["storage_failure"])
return
proc addPeer*(
@ -157,7 +159,7 @@ proc addPeer*(
trace "Adding peer to manager",
peerId = remotePeerInfo.peerId, addresses = remotePeerInfo.addrs, origin
waku_total_unique_peers.inc()
logos_delivery_total_unique_peers.inc()
# Add peer to storage. Entry will subsequently be updated with connectedness information
if not pm.storage.isNil:
@ -233,7 +235,7 @@ proc loadFromStorage(pm: PeerManager) {.gcsafe.} =
pm.storage.getAll(onData).isOkOr:
warn "loading peers from storage failed", err = error
waku_peers_errors.inc(labelValues = ["storage_load_failure"])
logos_delivery_peers_errors.inc(labelValues = ["storage_load_failure"])
return
trace "recovered peers from storage", amount = amount
@ -306,7 +308,7 @@ proc addServicePeer*(pm: PeerManager, remotePeerInfo: RemotePeerInfo, proto: str
info "Adding peer to service slots",
peerId = remotePeerInfo.peerId, addr = remotePeerInfo.addrs[0], service = proto
waku_service_peers.set(1, labelValues = [$proto, $remotePeerInfo.addrs[0]])
logos_delivery_service_peers.set(1, labelValues = [$proto, $remotePeerInfo.addrs[0]])
# Set peer for service slot
pm.serviceSlots[proto] = remotePeerInfo
@ -361,8 +363,8 @@ proc connectPeer*(
if not deadline.finished():
await deadline.cancelAndWait()
waku_peers_dials.inc(labelValues = ["successful"])
waku_node_conns_initiated.inc(labelValues = [source])
logos_delivery_peers_dials.inc(labelValues = ["successful"])
logos_delivery_node_conns_initiated.inc(labelValues = [source])
peerStore[NumberFailedConnBook][peerId] = 0
@ -377,7 +379,7 @@ proc connectPeer*(
peerId = peerId,
reason = reasonFailed,
failedAttempts = peerStore[NumberFailedConnBook][peerId]
waku_peers_dials.inc(labelValues = [reasonFailed])
logos_delivery_peers_dials.inc(labelValues = [reasonFailed])
return false
@ -893,16 +895,16 @@ proc logAndMetrics(pm: PeerManager) {.async.} =
for proto in peerStore.getWakuProtos():
let (protoConnsIn, protoConnsOut) = pm.connectedPeers(proto)
let (protoStreamsIn, protoStreamsOut) = pm.getNumStreams(proto)
waku_connected_peers.set(
logos_delivery_connected_peers.set(
protoConnsIn.len.float64, labelValues = [$Direction.In, proto]
)
waku_connected_peers.set(
logos_delivery_connected_peers.set(
protoConnsOut.len.float64, labelValues = [$Direction.Out, proto]
)
waku_streams_peers.set(
logos_delivery_streams_peers.set(
protoStreamsIn.float64, labelValues = [$Direction.In, proto]
)
waku_streams_peers.set(
logos_delivery_streams_peers.set(
protoStreamsOut.float64, labelValues = [$Direction.Out, proto]
)
@ -914,7 +916,9 @@ proc logAndMetrics(pm: PeerManager) {.async.} =
let agent = peerStore[AgentBook][peerId]
agentCounts[agent] = agentCounts.getOrDefault(agent, 0) + 1
for agent, count in agentCounts:
waku_connected_peers_per_agent.set(count.float64, labelValues = [$agent])
logos_delivery_connected_peers_per_agent.set(
count.float64, labelValues = [$agent]
)
for shard in pm.getShards().items:
# peers known for this shard
@ -924,7 +928,7 @@ proc logAndMetrics(pm: PeerManager) {.async.} =
# keep only those that are physically connected right now
let connectedInShard = shardPeers.filterIt(connectedPeerIds.contains(it.peerId))
waku_connected_peers_per_shard.set(
logos_delivery_connected_peers_per_shard.set(
connectedInShard.len.float64, labelValues = [$shard]
)
@ -1243,7 +1247,9 @@ proc new*(
var peerStore = pm.switch.peerStore
proc peerStoreChanged(peerId: PeerId) {.gcsafe.} =
waku_peer_store_size.set(toSeq(peerStore[AddressBook].book.keys).len.int64)
logos_delivery_peer_store_size.set(
toSeq(peerStore[AddressBook].book.keys).len.int64
)
pm.switch.addPeerEventHandler(peerHook, PeerEventKind.Joined)
pm.switch.addPeerEventHandler(peerHook, PeerEventKind.Left)

4
logos_delivery/waku/node/subscription_manager.nim
View File

@ -45,8 +45,8 @@ proc registerRelayHandler(
proc traceHandler(topic: PubsubTopic, msg: WakuMessage) {.async, gcsafe.} =
let msgSizeKB = msg.payload.len / 1000
waku_node_messages.inc(labelValues = ["relay"])
waku_histogram_message_size.observe(msgSizeKB)
logos_delivery_node_messages.inc(labelValues = ["relay"])
logos_delivery_histogram_message_size.observe(msgSizeKB)
proc filterHandler(topic: PubsubTopic, msg: WakuMessage) {.async, gcsafe.} =
if node.wakuFilter.isNil():

15
logos_delivery/waku/node/waku_metrics.nim
View File

@ -32,15 +32,16 @@ proc startMetricsLog*() =
# public in libp2p or do our own peer counting after all.
# track cumulative values
let freshErrorCount = parseAndAccumulate(waku_node_errors, cumulativeErrors)
let freshErrorCount =
parseAndAccumulate(logos_delivery_node_errors, cumulativeErrors)
let freshConnCount =
parseAndAccumulate(waku_node_conns_initiated, cumulativeConns)
parseAndAccumulate(logos_delivery_node_conns_initiated, cumulativeConns)
let totalMessages = collectorAsF64(waku_node_messages)
let storePeers = collectorAsF64(waku_store_peers)
let pxPeers = collectorAsF64(waku_px_peers)
let lightpushPeers = collectorAsF64(waku_lightpush_peers)
let filterPeers = collectorAsF64(waku_filter_peers)
let totalMessages = collectorAsF64(logos_delivery_node_messages)
let storePeers = collectorAsF64(logos_delivery_store_peers)
let pxPeers = collectorAsF64(logos_delivery_px_peers)
let lightpushPeers = collectorAsF64(logos_delivery_lightpush_peers)
let filterPeers = collectorAsF64(logos_delivery_filter_peers)
info "Total connections initiated", count = $freshConnCount
info "Total messages", count = totalMessages

2
logos_delivery/waku/node/waku_node.nim
View File

@ -594,7 +594,7 @@ proc start*(node: WakuNode) {.async.} =
## Starts a created Waku Node and
## all its mounted protocols.
waku_version.set(1, labelValues = [git_version])
logos_delivery_version.set(1, labelValues = [git_version])
info "Starting Waku node", version = git_version
var zeroPortPresent = false

18
logos_delivery/waku/node/waku_node/filter.nim
View File

@ -132,12 +132,12 @@ proc filterSubscribe*(
node.topicSubscriptionQueue.emit((kind: PubsubSub, topic: pubsubTopic.get()))
else:
error "failed filter v2 subscription", error = subRes.error
waku_node_errors.inc(labelValues = ["subscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["subscribe_filter_failure"])
return subRes
elif node.wakuAutoSharding.isNone():
error "Failed filter subscription, pubsub topic must be specified with static sharding"
waku_node_errors.inc(labelValues = ["subscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["subscribe_filter_failure"])
else:
# No pubsub topic, autosharding is used to deduce it
# but content topics must be well-formed for this
@ -163,7 +163,7 @@ proc filterSubscribe*(
if res.isErr():
error "failed filter subscription", error = res.error
waku_node_errors.inc(labelValues = ["subscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["subscribe_filter_failure"])
subRes = FilterSubscribeResult.err(res.error)
for pubsub, topics in topicMap.pairs:
@ -174,7 +174,7 @@ proc filterSubscribe*(
except CatchableError:
let errMsg = "exception in filterSubscribe: " & getCurrentExceptionMsg()
error "exception in filterSubscribe", error = getCurrentExceptionMsg()
waku_node_errors.inc(labelValues = ["subscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["subscribe_filter_failure"])
subRes =
FilterSubscribeResult.err(FilterSubscribeError.serviceUnavailable(errMsg))
@ -210,12 +210,12 @@ proc filterUnsubscribe*(
node.topicSubscriptionQueue.emit((kind: PubsubUnsub, topic: pubsubTopic.get()))
else:
error "failed filter unsubscription", error = unsubRes.error
waku_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
return unsubRes
elif node.wakuAutoSharding.isNone():
error "Failed filter un-subscription, pubsub topic must be specified with static sharding"
waku_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
else: # pubsubTopic.isNone
let topicMap = node.wakuAutoSharding
.get()
@ -239,7 +239,7 @@ proc filterUnsubscribe*(
if res.isErr():
error "failed filter unsubscription", error = res.error
waku_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
unsubRes = FilterSubscribeResult.err(res.error)
for pubsub, topics in topicMap.pairs:
@ -250,7 +250,7 @@ proc filterUnsubscribe*(
except CatchableError:
let errMsg = "exception in filterUnsubscribe: " & getCurrentExceptionMsg()
error "exception in filterUnsubscribe", error = getCurrentExceptionMsg()
waku_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
unsubRes =
FilterSubscribeResult.err(FilterSubscribeError.serviceUnavailable(errMsg))
@ -273,7 +273,7 @@ proc filterUnsubscribeAll*(
info "unsubscribed from all content-topic", peerId = remotePeer.peerId
else:
error "failed filter unsubscription from all content-topic", error = unsubRes.error
waku_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
logos_delivery_node_errors.inc(labelValues = ["unsubscribe_filter_failure"])
return unsubRes

2
logos_delivery/waku/node/waku_node/peer_exchange.nim
View File

@ -117,4 +117,4 @@ proc setPeerExchangePeer*(
return
node.peerManager.addPeer(remotePeer, PeerExchange)
waku_px_peers.inc()
logos_delivery_px_peers.inc()

2
logos_delivery/waku/node/waku_node/relay.nim
View File

@ -236,7 +236,7 @@ proc setRlnValidator*(
shareX = shareX,
shareY = shareY,
nullifier = nullifier
waku_rln_valid_messages_total.inc(labelValues = [topic])
logos_delivery_rln_valid_messages_total.inc(labelValues = [topic])
return pubsub.ValidationResult.Accept
of Invalid:
trace "message validity could not be verified, discarding",

10
logos_delivery/waku/rln/group_manager/on_chain/group_manager.nim
View File

@ -230,7 +230,7 @@ proc updateMemberCount*(
let nextFreeIndex = (await g.fetchNextFreeIndex()).valueOr:
return err("Failed to fetch next free index: " & error)
let memberCount = cast[int64](nextFreeIndex)
waku_rln_number_registered_memberships.set(float64(memberCount))
logos_delivery_rln_number_registered_memberships.set(float64(memberCount))
return ok()
proc refreshRoots(g: OnchainGroupManager): Future[void] {.async.} =
@ -549,7 +549,7 @@ method generateProof*(
external_nullifier: extNullifier,
)
waku_rln_proof_generation_duration_seconds.nanosecondTime:
logos_delivery_rln_proof_generation_duration_seconds.nanosecondTime:
let output = generateRlnProofWithWitness(
g.rlnInstance, witness, epoch, rlnIdentifier
).valueOr:
@ -557,8 +557,8 @@ method generateProof*(
info "Proof generated successfully", proof = output
waku_rln_remaining_proofs_per_epoch.dec()
waku_rln_total_generated_proofs.inc()
logos_delivery_rln_remaining_proofs_per_epoch.dec()
logos_delivery_rln_total_generated_proofs.inc()
return ok(output)
method verifyProof*(
@ -662,7 +662,7 @@ method init*(g: OnchainGroupManager): Future[GroupManagerResult[void]] {.async.}
)
if g.membershipIndex.isSome():
keystoreQuery.treeIndex = MembershipIndex(g.membershipIndex.get())
waku_rln_membership_credentials_import_duration_seconds.nanosecondTime:
logos_delivery_rln_membership_credentials_import_duration_seconds.nanosecondTime:
let keystoreCred = getMembershipCredentials(
path = g.keystorePath.get(),
password = g.keystorePassword.get(),

66
logos_delivery/waku/rln/protocol_metrics.nim
View File

@ -17,59 +17,65 @@ func generateBucketsForHistogram*(length: int): seq[float64] =
return buckets
declarePublicCounter(
waku_rln_messages_total, "number of messages seen by the rln relay"
logos_delivery_rln_messages_total, "number of messages seen by the rln relay"
)
declarePublicCounter(waku_rln_spam_messages_total, "number of spam messages detected")
declarePublicCounter(
waku_rln_invalid_messages_total, "number of invalid messages detected", ["type"]
logos_delivery_rln_spam_messages_total, "number of spam messages detected"
)
declarePublicCounter(
logos_delivery_rln_invalid_messages_total,
"number of invalid messages detected",
["type"],
)
# This metric will be useful in detecting the index of the root in the acceptable window of roots
declarePublicCounter(
waku_rln_valid_messages_total,
logos_delivery_rln_valid_messages_total,
"number of valid messages with their roots tracked",
["shard"],
)
declarePublicCounter(
waku_rln_errors_total,
logos_delivery_rln_errors_total,
"number of errors detected while operating the rln relay",
["type"],
)
declarePublicCounter(
waku_rln_proof_verification_total, "number of times the rln proofs are verified"
logos_delivery_rln_proof_verification_total,
"number of times the rln proofs are verified",
)
# this is a gauge so that we can set it based on the events we receive
declarePublicGauge(
waku_rln_number_registered_memberships,
logos_delivery_rln_number_registered_memberships,
"number of registered and active rln memberships",
)
# Timing metrics
declarePublicGauge(
waku_rln_proof_verification_duration_seconds, "time taken to verify a proof"
logos_delivery_rln_proof_verification_duration_seconds, "time taken to verify a proof"
)
declarePublicGauge(
waku_rln_proof_generation_duration_seconds, "time taken to generate a proof"
logos_delivery_rln_proof_generation_duration_seconds, "time taken to generate a proof"
)
declarePublicGauge(
waku_rln_instance_creation_duration_seconds, "time taken to create an rln instance"
logos_delivery_rln_instance_creation_duration_seconds,
"time taken to create an rln instance",
)
declarePublicGauge(
waku_rln_membership_insertion_duration_seconds,
logos_delivery_rln_membership_insertion_duration_seconds,
"time taken to process a new membership registration",
)
declarePublicGauge(
waku_rln_membership_credentials_import_duration_seconds,
logos_delivery_rln_membership_credentials_import_duration_seconds,
"time taken to import membership credentials",
)
declarePublicGauge(
waku_rln_remaining_proofs_per_epoch,
logos_delivery_rln_remaining_proofs_per_epoch,
"number of proofs remaining to be generated for the current epoch",
)
declarePublicGauge(
waku_rln_total_generated_proofs,
logos_delivery_rln_total_generated_proofs,
"total number of proofs generated since the node started",
)
@ -92,25 +98,29 @@ proc getRlnMetricsLogger*(): RLNMetricsLogger =
logMetrics = proc() =
{.gcsafe.}:
let freshErrorCount =
parseAndAccumulate(waku_rln_errors_total, cumulativeErrors)
parseAndAccumulate(logos_delivery_rln_errors_total, cumulativeErrors)
let freshMsgCount =
parseAndAccumulate(waku_rln_messages_total, cumulativeMessages)
let freshSpamCount =
parseAndAccumulate(waku_rln_spam_messages_total, cumulativeSpamMessages)
let freshInvalidMsgCount =
parseAndAccumulate(waku_rln_invalid_messages_total, cumulativeInvalidMessages)
let freshValidMsgCount =
parseAndAccumulate(waku_rln_valid_messages_total, cumulativeValidMessages)
let freshProofsVerifiedCount = parseAndAccumulate(
waku_rln_proof_verification_total, cumulativeProofsVerified
parseAndAccumulate(logos_delivery_rln_messages_total, cumulativeMessages)
let freshSpamCount = parseAndAccumulate(
logos_delivery_rln_spam_messages_total, cumulativeSpamMessages
)
let freshInvalidMsgCount = parseAndAccumulate(
logos_delivery_rln_invalid_messages_total, cumulativeInvalidMessages
)
let freshValidMsgCount = parseAndAccumulate(
logos_delivery_rln_valid_messages_total, cumulativeValidMessages
)
let freshProofsVerifiedCount = parseAndAccumulate(
logos_delivery_rln_proof_verification_total, cumulativeProofsVerified
)
let freshProofsGeneratedCount = parseAndAccumulate(
logos_delivery_rln_total_generated_proofs, cumulativeProofsGenerated
)
let freshProofsGeneratedCount =
parseAndAccumulate(waku_rln_total_generated_proofs, cumulativeProofsGenerated)
let freshProofsRemainingCount = parseAndAccumulate(
waku_rln_remaining_proofs_per_epoch, cumulativeProofsRemaining
logos_delivery_rln_remaining_proofs_per_epoch, cumulativeProofsRemaining
)
let freshRegisteredMemberCount = parseAndAccumulate(
waku_rln_number_registered_memberships, cumulativeRegisteredMember
logos_delivery_rln_number_registered_memberships, cumulativeRegisteredMember
)
info "Total messages", count = freshMsgCount

24
logos_delivery/waku/rln/rln.nim
View File

@ -61,7 +61,7 @@ proc validateMessage*(
return MessageValidationResult.Invalid
# track message count for metrics
waku_rln_messages_total.inc()
logos_delivery_rln_messages_total.inc()
# checks if the message's timestamp is within acceptable range
let currentTime = getTime().toUnixFloat()
@ -77,7 +77,7 @@ proc validateMessage*(
timeDiff = timeDiff,
maxTimestampGap = rlnPeer.rlnMaxTimestampGap,
contentTopic = msg.contentTopic
waku_rln_invalid_messages_total.inc(labelValues = ["invalid_timestamp"])
logos_delivery_rln_invalid_messages_total.inc(labelValues = ["invalid_timestamp"])
return MessageValidationResult.Invalid
let computedEpoch = rlnPeer.calcEpoch(messageTime)
@ -86,7 +86,7 @@ proc validateMessage*(
proofEpoch = fromEpoch(proof.epoch),
computedEpoch = fromEpoch(computedEpoch),
contentTopic = msg.contentTopic
waku_rln_invalid_messages_total.inc(labelValues = ["timestamp_mismatch"])
logos_delivery_rln_invalid_messages_total.inc(labelValues = ["timestamp_mismatch"])
return MessageValidationResult.Invalid
let rootValidationRes = await rlnPeer.groupManager.validateRoot(proof.merkleRoot)
@ -95,7 +95,7 @@ proc validateMessage*(
provided = proof.merkleRoot.inHex(),
validRoots = rlnPeer.groupManager.validRoots.mapIt(it.inHex()),
contentTopic = msg.contentTopic
waku_rln_invalid_messages_total.inc(labelValues = ["invalid_root"])
logos_delivery_rln_invalid_messages_total.inc(labelValues = ["invalid_root"])
return MessageValidationResult.Invalid
# verify the proof
@ -104,13 +104,13 @@ proc validateMessage*(
timestampBytes = toBytes(msg.timestamp.uint64)
input = concat(msg.payload, contentTopicBytes, @(timestampBytes))
waku_rln_proof_verification_total.inc()
waku_rln_proof_verification_duration_seconds.nanosecondTime:
logos_delivery_rln_proof_verification_total.inc()
logos_delivery_rln_proof_verification_duration_seconds.nanosecondTime:
let proofVerificationRes =
rlnPeer.groupManager.verifyProof(msg.toRLNSignal(), proof)
proofVerificationRes.isOkOr:
waku_rln_errors_total.inc(labelValues = ["proof_verification"])
logos_delivery_rln_errors_total.inc(labelValues = ["proof_verification"])
warn "invalid message: proof verification failed",
payloadLen = msg.payload.len, contentTopic = msg.contentTopic
return MessageValidationResult.Invalid
@ -119,22 +119,22 @@ proc validateMessage*(
# invalid proof
warn "invalid message: invalid proof",
payloadLen = msg.payload.len, contentTopic = msg.contentTopic
waku_rln_invalid_messages_total.inc(labelValues = ["invalid_proof"])
logos_delivery_rln_invalid_messages_total.inc(labelValues = ["invalid_proof"])
return MessageValidationResult.Invalid
# check if double messaging has happened
let proofMetadata = proof.extractMetadata().valueOr:
waku_rln_errors_total.inc(labelValues = ["proof_metadata_extraction"])
logos_delivery_rln_errors_total.inc(labelValues = ["proof_metadata_extraction"])
return MessageValidationResult.Invalid
let msgEpoch = proof.epoch
let hasDup = rlnPeer.hasDuplicate(msgEpoch, proofMetadata)
if hasDup.isErr():
waku_rln_errors_total.inc(labelValues = ["duplicate_check"])
logos_delivery_rln_errors_total.inc(labelValues = ["duplicate_check"])
elif hasDup.value == true:
trace "invalid message: message is spam",
payloadLen = msg.payload.len, contentTopic = msg.contentTopic
waku_rln_spam_messages_total.inc()
logos_delivery_rln_spam_messages_total.inc()
return MessageValidationResult.Spam
trace "message is valid",
@ -167,7 +167,7 @@ proc monitorEpochs(rln: Rln) {.async.} =
while true:
try:
if rln.groupManager.userMessageLimit.isSome():
waku_rln_remaining_proofs_per_epoch.set(
logos_delivery_rln_remaining_proofs_per_epoch.set(
rln.groupManager.userMessageLimit.get().float64
)
else:

2
logos_delivery/waku/rln/rln/wrappers.nim
View File

@ -145,7 +145,7 @@ proc createRLNInstanceLocal(): Result[ptr RlnRaw, string] =
proc createRLNInstance*(): Result[ptr RlnRaw, string] =
## Wraps createRLNInstanceLocal with metrics timing.
var res: Result[ptr RlnRaw, string]
waku_rln_instance_creation_duration_seconds.nanosecondTime:
logos_delivery_rln_instance_creation_duration_seconds.nanosecondTime:
res = createRLNInstanceLocal()
return res

18
logos_delivery/waku/waku_archive/archive.nim
View File

@ -107,7 +107,7 @@ proc handleMessage*(
msgTimestamp = msg.timestamp
self.validator(msg).isOkOr:
waku_archive_errors.inc(labelValues = [error])
logos_delivery_archive_errors.inc(labelValues = [error])
trace "invalid message",
msg_hash = msgHashHex,
pubsubTopic = pubsubTopic,
@ -119,7 +119,7 @@ proc handleMessage*(
let insertStartTime = getTime().toUnixFloat()
(await self.driver.put(msgHash, pubsubTopic, msg)).isOkOr:
waku_archive_errors.inc(labelValues = [insertFailure])
logos_delivery_archive_errors.inc(labelValues = [insertFailure])
trace "failed to insert message",
msg_hash = msgHashHex,
pubsubTopic = pubsubTopic,
@ -129,12 +129,12 @@ proc handleMessage*(
return
let insertDuration = getTime().toUnixFloat() - insertStartTime
waku_archive_insert_duration_seconds.observe(insertDuration)
logos_delivery_archive_insert_duration_seconds.observe(insertDuration)
let shard = RelayShard.parseStaticSharding(pubsubTopic).valueOr:
DefaultRelayShard
waku_archive_messages_per_shard.inc(labelValues = [$shard.shardId])
logos_delivery_archive_messages_per_shard.inc(labelValues = [$shard.shardId])
trace "message archived",
msg_hash = msgHashHex,
@ -162,7 +162,7 @@ proc syncMessageIngress*(
let insertStartTime = getTime().toUnixFloat()
(await self.driver.put(msgHash, pubsubTopic, msg)).isOkOr:
waku_archive_errors.inc(labelValues = [insertFailure])
logos_delivery_archive_errors.inc(labelValues = [insertFailure])
trace "failed to insert message in in syncMessageIngress",
msg_hash = msgHashHex,
pubsubTopic = pubsubTopic,
@ -172,7 +172,7 @@ proc syncMessageIngress*(
return err(error)
let insertDuration = getTime().toUnixFloat() - insertStartTime
waku_archive_insert_duration_seconds.observe(insertDuration)
logos_delivery_archive_insert_duration_seconds.observe(insertDuration)
trace "message archived in syncMessageIngress",
msg_hash = msgHashHex,
@ -227,7 +227,7 @@ proc findMessages*(
).valueOr:
return err(ArchiveError(kind: ArchiveErrorKind.DRIVER_ERROR, cause: error))
let queryDuration = getTime().toUnixFloat() - queryStartTime
waku_archive_query_duration_seconds.observe(queryDuration)
logos_delivery_archive_query_duration_seconds.observe(queryDuration)
var hashes = newSeq[WakuMessageHash]()
var messages = newSeq[WakuMessage]()
@ -269,7 +269,7 @@ proc periodicRetentionPolicy(self: WakuArchive) {.async.} =
for policy in self.retentionPolicies:
info "executing message retention policy", policy = $policy
(await policy.execute(self.driver)).isOkOr:
waku_archive_errors.inc(labelValues = [retPolicyFailure])
logos_delivery_archive_errors.inc(labelValues = [retPolicyFailure])
error "failed execution of retention policy", policy = $policy, error = error
await sleepAsync(WakuArchiveDefaultRetentionPolicyIntervalWhenError)
## in case of error, let's try again faster
@ -285,7 +285,7 @@ proc periodicMetricReport(self: WakuArchive) {.async.} =
error = countRes.error
else:
let count = countRes.get()
waku_archive_messages.set(count, labelValues = ["stored"])
logos_delivery_archive_messages.set(count, labelValues = ["stored"])
await sleepAsync(WakuArchiveDefaultMetricsReportInterval)

13
logos_delivery/waku/waku_archive/archive_metrics.nim
View File

@ -2,13 +2,16 @@
import metrics
declarePublicGauge waku_archive_messages, "number of historical messages", ["type"]
declarePublicGauge waku_archive_messages_per_shard,
declarePublicGauge logos_delivery_archive_messages,
"number of historical messages", ["type"]
declarePublicGauge logos_delivery_archive_messages_per_shard,
"number of historical messages per shard ", ["shard"]
declarePublicCounter waku_archive_errors, "number of store protocol errors", ["type"]
declarePublicHistogram waku_archive_insert_duration_seconds,
declarePublicCounter logos_delivery_archive_errors,
"number of store protocol errors", ["type"]
declarePublicHistogram logos_delivery_archive_insert_duration_seconds,
"message insertion duration"
declarePublicHistogram waku_archive_query_duration_seconds, "history query duration"
declarePublicHistogram logos_delivery_archive_query_duration_seconds,
"history query duration"
# Error types (metric label values)
const

4
logos_delivery/waku/waku_archive/driver/postgres_driver/postgres_driver.nim
View File

@ -20,7 +20,7 @@ import
logScope:
topics = "postgres driver"
declarePublicGauge postgres_payload_size_bytes,
declarePublicGauge logos_delivery_postgres_payload_size_bytes,
"Payload size in bytes of correctly stored messages"
logScope:
@ -343,7 +343,7 @@ method put*(
)
if ret.isOk():
postgres_payload_size_bytes.set(message.payload.len)
logos_delivery_postgres_payload_size_bytes.set(message.payload.len)
return ret
method getAllMessages*(

16
logos_delivery/waku/waku_filter_v2/client.nim
View File

@ -45,12 +45,12 @@ proc sendSubscribeRequest(
connOpt = await wfc.peerManager.dialPeer(servicePeer, WakuFilterSubscribeCodec)
if connOpt.isNone():
trace "Failed to dial filter service peer", servicePeer
waku_filter_errors.inc(labelValues = [dialFailure])
logos_delivery_filter_errors.inc(labelValues = [dialFailure])
return err(FilterSubscribeError.peerDialFailure($servicePeer))
except CatchableError:
let errMsg = "failed to dialPeer: " & getCurrentExceptionMsg()
trace "failed to dialPeer", error = getCurrentExceptionMsg()
waku_filter_errors.inc(labelValues = [errMsg])
logos_delivery_filter_errors.inc(labelValues = [errMsg])
return err(FilterSubscribeError.badResponse(errMsg))
let connection = connOpt.get()
@ -64,7 +64,7 @@ proc sendSubscribeRequest(
let errMsg =
"exception in waku_filter_v2 client writeLP: " & getCurrentExceptionMsg()
trace "exception in waku_filter_v2 client writeLP", error = getCurrentExceptionMsg()
waku_filter_errors.inc(labelValues = [errMsg])
logos_delivery_filter_errors.inc(labelValues = [errMsg])
return err(FilterSubscribeError.badResponse(errMsg))
var respBuf: seq[byte]
@ -74,24 +74,24 @@ proc sendSubscribeRequest(
let errMsg =
"exception in waku_filter_v2 client readLp: " & getCurrentExceptionMsg()
trace "exception in waku_filter_v2 client readLp", error = getCurrentExceptionMsg()
waku_filter_errors.inc(labelValues = [errMsg])
logos_delivery_filter_errors.inc(labelValues = [errMsg])
return err(FilterSubscribeError.badResponse(errMsg))
let response = FilterSubscribeResponse.decode(respBuf).valueOr:
trace "Failed to decode filter subscribe response", servicePeer
waku_filter_errors.inc(labelValues = [decodeRpcFailure])
logos_delivery_filter_errors.inc(labelValues = [decodeRpcFailure])
return err(FilterSubscribeError.badResponse(decodeRpcFailure))
# DOS protection rate limit checks does not know about request id
if response.statusCode != FilterSubscribeErrorKind.TOO_MANY_REQUESTS.uint32 and
response.requestId != filterSubscribeRequest.requestId:
trace "Filter subscribe response requestId mismatch", servicePeer, response
waku_filter_errors.inc(labelValues = [requestIdMismatch])
logos_delivery_filter_errors.inc(labelValues = [requestIdMismatch])
return err(FilterSubscribeError.badResponse(requestIdMismatch))
if response.statusCode != 200:
trace "Filter subscribe error response", servicePeer, response
waku_filter_errors.inc(labelValues = [errorResponse])
logos_delivery_filter_errors.inc(labelValues = [errorResponse])
let cause =
if response.statusDesc.isSome():
response.statusDesc.get()
@ -188,7 +188,7 @@ proc initProtocolHandler(wfc: WakuFilterClient) =
let msgPush = MessagePush.decode(buf).valueOr:
error "Failed to decode message push", peerId = conn.peerId, error = $error
waku_filter_errors.inc(labelValues = [decodeRpcFailure])
logos_delivery_filter_errors.inc(labelValues = [decodeRpcFailure])
return
let msg_hash =

18
logos_delivery/waku/waku_filter_v2/protocol.nim
View File

@ -130,7 +130,7 @@ proc handleSubscribeRequest*(
wf: WakuFilter, peerId: PeerId, request: FilterSubscribeRequest
): Future[FilterSubscribeResponse] {.async.} =
info "received filter subscribe request", peerId = peerId, request = request
waku_filter_requests.inc(labelValues = [$request.filterSubscribeType])
logos_delivery_filter_requests.inc(labelValues = [$request.filterSubscribeType])
var subscribeResult: FilterSubscribeResult
@ -154,7 +154,7 @@ proc handleSubscribeRequest*(
requestDuration = Moment.now() - requestStartTime
requestDurationSec = requestDuration.milliseconds.float / 1000
# Duration in seconds with millisecond precision floating point
waku_filter_request_duration_seconds.observe(
logos_delivery_filter_request_duration_seconds.observe(
requestDurationSec, labelValues = [$request.filterSubscribeType]
)
@ -181,7 +181,7 @@ proc pushToPeer(
await stream.writeLp(buffer)
info "published successful", peerId = shortLog(peerId), stream
waku_service_network_bytes.inc(
logos_delivery_service_network_bytes.inc(
amount = buffer.len().int64, labelValues = [WakuFilterPushCodec, "out"]
)
@ -237,7 +237,7 @@ proc maintainSubscriptions*(wf: WakuFilter) {.async.} =
wf.subscriptions.cleanUp()
## Periodic report of number of subscriptions
waku_filter_subscriptions.set(wf.subscriptions.peersSubscribed.len.float64)
logos_delivery_filter_subscriptions.set(wf.subscriptions.peersSubscribed.len.float64)
const MessagePushTimeout = 20.seconds
proc handleMessage*(
@ -272,7 +272,7 @@ proc handleMessage*(
msg_hash = msgHash,
numPeers = subscribedPeers.len,
target_peer_ids = subscribedPeers.mapIt(shortLog(it))
waku_filter_errors.inc(labelValues = [pushTimeoutFailure])
logos_delivery_filter_errors.inc(labelValues = [pushTimeoutFailure])
else:
notice "pushed message succesfully to all subscribers",
pubsubTopic = pubsubTopic,
@ -285,7 +285,9 @@ proc handleMessage*(
handleMessageDuration = Moment.now() - handleMessageStartTime
handleMessageDurationSec = handleMessageDuration.milliseconds.float / 1000
# Duration in seconds with millisecond precision floating point
waku_filter_handle_message_duration_seconds.observe(handleMessageDurationSec)
logos_delivery_filter_handle_message_duration_seconds.observe(
handleMessageDurationSec
)
proc initProtocolHandler(wf: WakuFilter) =
proc handler(conn: Connection, proto: string) {.async: (raises: [CancelledError]).} =
@ -306,14 +308,14 @@ proc initProtocolHandler(wf: WakuFilter) =
remote_peer_id = conn.peerId, error = getCurrentExceptionMsg()
return
waku_service_network_bytes.inc(
logos_delivery_service_network_bytes.inc(
amount = buf.len().int64, labelValues = [WakuFilterSubscribeCodec, "in"]
)
let request = FilterSubscribeRequest.decode(buf).valueOr:
error "failed to decode filter subscribe request",
peer_id = conn.peerId, err = error
waku_filter_errors.inc(labelValues = [decodeRpcFailure])
logos_delivery_filter_errors.inc(labelValues = [decodeRpcFailure])
return
try:

12
logos_delivery/waku/waku_filter_v2/protocol_metrics.nim
View File

@ -4,13 +4,15 @@ import metrics
export metrics
declarePublicCounter waku_filter_errors, "number of filter protocol errors", ["type"]
declarePublicCounter waku_filter_requests,
declarePublicCounter logos_delivery_filter_errors,
"number of filter protocol errors", ["type"]
declarePublicCounter logos_delivery_filter_requests,
"number of filter subscribe requests received", ["type"]
declarePublicGauge waku_filter_subscriptions, "number of subscribed filter clients"
declarePublicHistogram waku_filter_request_duration_seconds,
declarePublicGauge logos_delivery_filter_subscriptions,
"number of subscribed filter clients"
declarePublicHistogram logos_delivery_filter_request_duration_seconds,
"duration of filter subscribe requests", ["type"]
declarePublicHistogram waku_filter_handle_message_duration_seconds,
declarePublicHistogram logos_delivery_filter_handle_message_duration_seconds,
"duration to push message to filter subscribers",
buckets = [
0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0,

4
logos_delivery/waku/waku_lightpush/client.nim
View File

@ -38,7 +38,7 @@ proc sendPushRequest(
): Future[WakuLightPushResult] {.async.} =
let connection = conn.valueOr:
(await wl.peerManager.dialPeer(peer, WakuLightPushCodec)).valueOr:
waku_lightpush_v3_errors.inc(labelValues = [dialFailure])
logos_delivery_lightpush_v3_errors.inc(labelValues = [dialFailure])
return lighpushErrorResult(
LightPushErrorCode.NO_PEERS_TO_RELAY,
dialFailure & ": " & $peer & " is not accessible",
@ -60,7 +60,7 @@ proc sendPushRequest(
let response = LightpushResponse.decode(buffer).valueOr:
error "failed to decode response"
waku_lightpush_v3_errors.inc(labelValues = [decodeRpcFailure])
logos_delivery_lightpush_v3_errors.inc(labelValues = [decodeRpcFailure])
return lightpushResultInternalError(decodeRpcFailure)
if response.requestId != req.requestId and

8
logos_delivery/waku/waku_lightpush/protocol.nim
View File

@ -55,7 +55,7 @@ proc handleRequest(
(code: LightPushErrorCode.BAD_REQUEST, desc: Opt.some(msg))
)
waku_lightpush_v3_messages.inc(labelValues = ["PushRequest"])
logos_delivery_lightpush_v3_messages.inc(labelValues = ["PushRequest"])
let msg_hash = pubsubTopic.computeMessageHash(pushRequest.message).to0xHex()
notice "handling lightpush request",
@ -78,7 +78,7 @@ proc handleRequest*(
let desc = decodeRpcFailure & ": " & $error
error "failed to decode Lightpush request", error = desc
let errorCode = LightPushErrorCode.BAD_REQUEST
waku_lightpush_v3_errors.inc(labelValues = [$errorCode])
logos_delivery_lightpush_v3_errors.inc(labelValues = [$errorCode])
return LightPushResponse(
requestId: "N/A", # due to decode failure we don't know requestId
statusCode: errorCode,
@ -87,7 +87,7 @@ proc handleRequest*(
let relayPeerCount = (await wl.handleRequest(peerId, request)).valueOr:
let desc = error.desc
waku_lightpush_v3_errors.inc(labelValues = [$error.code])
logos_delivery_lightpush_v3_errors.inc(labelValues = [$error.code])
error "failed to push message", error = desc.get("")
return LightPushResponse(
requestId: request.requestId, statusCode: error.code, statusDesc: desc
@ -114,7 +114,7 @@ proc initProtocolHandler(wl: WakuLightPush) =
error "lightpush read stream failed", error = getCurrentExceptionMsg()
return
waku_service_network_bytes.inc(
logos_delivery_service_network_bytes.inc(
amount = buffer.len().int64, labelValues = [WakuLightPushCodec, "in"]
)

4
logos_delivery/waku/waku_lightpush/protocol_metrics.nim
View File

@ -2,9 +2,9 @@
import metrics
declarePublicCounter waku_lightpush_v3_errors,
declarePublicCounter logos_delivery_lightpush_v3_errors,
"number of lightpush protocol errors", ["type"]
declarePublicCounter waku_lightpush_v3_messages,
declarePublicCounter logos_delivery_lightpush_v3_messages,
"number of lightpush messages received", ["type"]
# Error types (metric label values)

6
logos_delivery/waku/waku_lightpush_legacy/client.nim
View File

@ -30,7 +30,7 @@ proc sendPushRequest(
): Future[WakuLightPushResult[void]] {.async, gcsafe.} =
let connOpt = await wl.peerManager.dialPeer(peer, WakuLegacyLightPushCodec)
if connOpt.isNone():
waku_lightpush_errors.inc(labelValues = [dialFailure])
logos_delivery_lightpush_errors.inc(labelValues = [dialFailure])
return err(dialFailure)
let connection = connOpt.get()
@ -48,11 +48,11 @@ proc sendPushRequest(
let pushResponseRes = PushRPC.decode(buffer).valueOr:
error "failed to decode response"
waku_lightpush_errors.inc(labelValues = [decodeRpcFailure])
logos_delivery_lightpush_errors.inc(labelValues = [decodeRpcFailure])
return err(decodeRpcFailure)
if pushResponseRes.response.isNone():
waku_lightpush_errors.inc(labelValues = [emptyResponseBodyFailure])
logos_delivery_lightpush_errors.inc(labelValues = [emptyResponseBodyFailure])
return err(emptyResponseBodyFailure)
let response = pushResponseRes.response.get()

6
logos_delivery/waku/waku_lightpush_legacy/protocol.nim
View File

@ -44,7 +44,7 @@ proc handleRequest*(
pubSubTopic = request.get().pubSubTopic
message = request.get().message
let msg_hash = pubsubTopic.computeMessageHash(message).to0xHex()
waku_lightpush_messages.inc(labelValues = ["PushRequest"])
logos_delivery_lightpush_messages.inc(labelValues = ["PushRequest"])
notice "handling legacy lightpush request",
my_peer_id = wl.peerManager.switch.peerInfo.peerId,
@ -60,7 +60,7 @@ proc handleRequest*(
pushResponseInfo = (if isSuccess: "OK" else: handleRes.error)
if not isSuccess:
waku_lightpush_errors.inc(labelValues = [pushResponseInfo])
logos_delivery_lightpush_errors.inc(labelValues = [pushResponseInfo])
error "failed to push message", error = pushResponseInfo
let response = PushResponse(isSuccess: isSuccess, info: Opt.some(pushResponseInfo))
let rpc = PushRPC(requestId: requestId, response: Opt.some(response))
@ -80,7 +80,7 @@ proc initProtocolHandler(wl: WakuLegacyLightPush) =
error "lightpush legacy read stream failed", error = getCurrentExceptionMsg()
return
waku_service_network_bytes.inc(
logos_delivery_service_network_bytes.inc(
amount = buffer.len().int64, labelValues = [WakuLegacyLightPushCodec, "in"]
)

4
logos_delivery/waku/waku_lightpush_legacy/protocol_metrics.nim
View File

@ -2,9 +2,9 @@
import metrics
declarePublicCounter waku_lightpush_errors,
declarePublicCounter logos_delivery_lightpush_errors,
"number of lightpush protocol errors", ["type"]
declarePublicCounter waku_lightpush_messages,
declarePublicCounter logos_delivery_lightpush_messages,
"number of lightpush messages received", ["type"]
# Error types (metric label values)

2
logos_delivery/waku/waku_lightpush_legacy/self_req_handler.nim
View File

@ -36,7 +36,7 @@ proc handleSelfLightPushRequest*(
let respRpc = await self.handleRequest(selfPeerId, rpc.encode().buffer)
if respRpc.response.isNone():
waku_lightpush_errors.inc(labelValues = [emptyResponseBodyFailure])
logos_delivery_lightpush_errors.inc(labelValues = [emptyResponseBodyFailure])
return err(emptyResponseBodyFailure)
let response = respRpc.response.get()

11
logos_delivery/waku/waku_peer_exchange/client.nim
View File

@ -10,9 +10,10 @@ import
from logos_delivery/waku/waku_core/codecs import WakuPeerExchangeCodec
declarePublicGauge waku_px_peers_received_total,
declarePublicGauge logos_delivery_px_peers_received_total,
"number of ENRs received via peer exchange"
declarePublicCounter waku_px_client_errors, "number of peer exchange errors", ["type"]
declarePublicCounter logos_delivery_px_client_errors,
"number of peer exchange errors", ["type"]
logScope:
topics = "waku peer_exchange client"
@ -37,7 +38,9 @@ proc request*(
buffer = await conn.readLp(DefaultMaxRpcSize.int)
except CatchableError as exc:
error "exception when handling peer exchange request", error = exc.msg
waku_px_client_errors.inc(labelValues = ["error_sending_or_receiving_px_req"])
logos_delivery_px_client_errors.inc(
labelValues = ["error_sending_or_receiving_px_req"]
)
callResult = (
status_code: PeerExchangeResponseStatusCode.SERVICE_UNAVAILABLE,
status_desc: Opt.some($exc.msg),
@ -97,7 +100,7 @@ proc request*(
): Future[WakuPeerExchangeResult[PeerExchangeResponse]] {.async: (raises: []).} =
let peerOpt = wpx.peerManager.selectPeer(WakuPeerExchangeCodec)
if peerOpt.isNone():
waku_px_client_errors.inc(labelValues = [peerNotFoundFailure])
logos_delivery_px_client_errors.inc(labelValues = [peerNotFoundFailure])
info "peer exchange request could not be made as no peer exchange peers found"
return err(
(

17
logos_delivery/waku/waku_peer_exchange/protocol.nim
View File

@ -21,10 +21,11 @@ import
from ../waku_core/codecs import WakuPeerExchangeCodec
export WakuPeerExchangeCodec
declarePublicGauge waku_px_peers_received_unknown,
declarePublicGauge logos_delivery_px_peers_received_unknown,
"number of previously unknown ENRs received via peer exchange"
declarePublicCounter waku_px_errors, "number of peer exchange errors", ["type"]
declarePublicCounter waku_px_peers_sent,
declarePublicCounter logos_delivery_px_errors,
"number of peer exchange errors", ["type"]
declarePublicCounter logos_delivery_px_peers_sent,
"number of ENRs sent to peer exchange requesters"
logScope:
@ -47,7 +48,7 @@ proc respond(
# Remote closed the stream before we responded - expected during peer churn.
debug "peer exchange response not delivered: stream closed",
peerId = conn.peerId, error = exc.msg
waku_px_errors.inc(labelValues = [streamClosedFailure])
logos_delivery_px_errors.inc(labelValues = [streamClosedFailure])
return err(
(
status_code: PeerExchangeResponseStatusCode.DIAL_FAILURE,
@ -71,7 +72,7 @@ proc respondError(
# Remote closed the stream before we responded - expected during peer churn.
debug "peer exchange error response not delivered: stream closed",
peerId = conn.peerId, error = exc.msg
waku_px_errors.inc(labelValues = [streamClosedFailure])
logos_delivery_px_errors.inc(labelValues = [streamClosedFailure])
return err(
(
status_code: PeerExchangeResponseStatusCode.SERVICE_UNAVAILABLE,
@ -137,11 +138,11 @@ proc initProtocolHandler(wpx: WakuPeerExchange) =
# no point responding on a dead stream.
debug "peer exchange request not received: stream closed",
peerId = conn.peerId, error = exc.msg
waku_px_errors.inc(labelValues = [streamClosedFailure])
logos_delivery_px_errors.inc(labelValues = [streamClosedFailure])
return
let decBuf = PeerExchangeRpc.decode(buffer).valueOr:
waku_px_errors.inc(labelValues = [decodeRpcFailure])
logos_delivery_px_errors.inc(labelValues = [decodeRpcFailure])
error "Failed to decode PeerExchange request", error = $error
(
@ -157,7 +158,7 @@ proc initProtocolHandler(wpx: WakuPeerExchange) =
info "peer exchange request received"
trace "px enrs to respond", enrs = $enrs
(await wpx.respond(enrs, conn)).isErrOr:
waku_px_peers_sent.inc(enrs.len().int64())
logos_delivery_px_peers_sent.inc(enrs.len().int64())
do:
defer:
# close, no data is expected

26
logos_delivery/waku/waku_relay/protocol.nim
View File

@ -39,24 +39,24 @@ type ShardMetrics = object
logScope:
topics = "waku relay"
declareCounter waku_relay_network_bytes,
declareCounter logos_delivery_relay_network_bytes,
"total traffic per topic, distinct gross/net and direction",
labels = ["topic", "type", "direction"]
declarePublicGauge(
waku_relay_total_msg_bytes_per_shard,
logos_delivery_relay_total_msg_bytes_per_shard,
"total length of messages seen per shard",
labels = ["shard"],
)
declarePublicGauge(
waku_relay_max_msg_bytes_per_shard,
logos_delivery_relay_max_msg_bytes_per_shard,
"Maximum length of messages seen per shard",
labels = ["shard"],
)
declarePublicGauge(
waku_relay_avg_msg_bytes_per_shard,
logos_delivery_relay_avg_msg_bytes_per_shard,
"Average length of messages seen per shard",
labels = ["shard"],
)
@ -245,11 +245,17 @@ proc logMessageInfo*(
shardMetrics.avgSize = shardMetrics.sizeSum / shardMetrics.count
w.msgMetricsPerShard[topic] = shardMetrics
waku_relay_max_msg_bytes_per_shard.set(shardMetrics.maxSize, labelValues = [topic])
logos_delivery_relay_max_msg_bytes_per_shard.set(
shardMetrics.maxSize, labelValues = [topic]
)
waku_relay_avg_msg_bytes_per_shard.set(shardMetrics.avgSize, labelValues = [topic])
logos_delivery_relay_avg_msg_bytes_per_shard.set(
shardMetrics.avgSize, labelValues = [topic]
)
waku_relay_total_msg_bytes_per_shard.set(shardMetrics.sizeSum, labelValues = [topic])
logos_delivery_relay_total_msg_bytes_per_shard.set(
shardMetrics.sizeSum, labelValues = [topic]
)
proc initRelayObservers(w: WakuRelay) =
proc decodeRpcMessageInfo(
@ -287,13 +293,13 @@ proc initRelayObservers(w: WakuRelay) =
onRecv: bool,
) =
if onRecv:
waku_relay_network_bytes.inc(
logos_delivery_relay_network_bytes.inc(
msgSize.int64, labelValues = [pubsub_topic, "gross", "in"]
)
else:
# sent traffic can only be "net"
# TODO: If we can measure unsuccessful sends would mean a possible distinction between gross/net
waku_relay_network_bytes.inc(
logos_delivery_relay_network_bytes.inc(
msgSize.int64, labelValues = [pubsub_topic, "net", "out"]
)
@ -608,7 +614,7 @@ proc subscribe*(w: WakuRelay, pubsubTopic: PubsubTopic, handler: WakuRelayHandle
return fut
# this subscription handler is called once for every validated message
# that will be relayed, hence this is the place we can count net incoming traffic
waku_relay_network_bytes.inc(
logos_delivery_relay_network_bytes.inc(
data.len.int64 + pubsubTopic.len.int64, labelValues = [pubsubTopic, "net", "in"]
)

4
logos_delivery/waku/waku_rendezvous/client.nim
View File

@ -21,7 +21,7 @@ import
logScope:
topics = "waku rendezvous client"
declarePublicCounter rendezvousPeerFoundTotal,
declarePublicCounter logos_delivery_rendezvous_peer_found,
"total number of peers found via rendezvous"
type WakuRendezVousClient* = ref object
@ -59,7 +59,7 @@ proc requestAll*(
trace "waku rendezvous client request got peers", count = records.len
for record in records:
if not self.switch.peerStore.peerExists(record.peerId):
rendezvousPeerFoundTotal.inc()
logos_delivery_rendezvous_peer_found.inc()
if record.mixKey.len == 0 or record.peerId == self.switch.peerInfo.peerId:
continue
trace "adding peer from rendezvous",

12
logos_delivery/waku/waku_store/client.nim
View File

@ -54,11 +54,11 @@ proc sendStoreRequest(
return err(StoreError(kind: ErrorCode.BAD_RESPONSE, cause: error.msg))
let res = StoreQueryResponse.decode(buf).valueOr:
waku_store_errors.inc(labelValues = [DecodeRpcFailure])
logos_delivery_store_errors.inc(labelValues = [DecodeRpcFailure])
return err(StoreError(kind: ErrorCode.BAD_RESPONSE, cause: DecodeRpcFailure))
if res.statusCode != uint32(StatusCode.SUCCESS):
waku_store_errors.inc(labelValues = [NoSuccessStatusCode])
logos_delivery_store_errors.inc(labelValues = [NoSuccessStatusCode])
return err(StoreError.new(res.statusCode, res.statusDesc))
if req.pubsubTopic.isSome():
@ -67,10 +67,10 @@ proc sendStoreRequest(
self.storeMsgMetricsPerShard[topic] = 0
self.storeMsgMetricsPerShard[topic] += float64(req.encode().buffer.len)
waku_relay_fleet_store_msg_size_bytes.inc(
logos_delivery_store_fleet_msg_size_bytes.inc(
self.storeMsgMetricsPerShard[topic], labelValues = [topic]
)
waku_relay_fleet_store_msg_count.inc(1.0, labelValues = [topic])
logos_delivery_store_fleet_msg_count.inc(1.0, labelValues = [topic])
return ok(res)
@ -81,7 +81,7 @@ proc query*(
return err(StoreError(kind: ErrorCode.BAD_REQUEST, cause: "invalid cursor"))
let connection = (await self.peerManager.dialPeer(peer, WakuStoreCodec)).valueOr:
waku_store_errors.inc(labelValues = [DialFailure])
logos_delivery_store_errors.inc(labelValues = [DialFailure])
return err(StoreError(kind: ErrorCode.PEER_DIAL_FAILURE, address: $peer))
@ -108,7 +108,7 @@ proc queryToAny*(
var lastError: StoreError
for peer in peersToTry:
let connection = (await self.peerManager.dialPeer(peer, WakuStoreCodec)).valueOr:
waku_store_errors.inc(labelValues = [DialFailure])
logos_delivery_store_errors.inc(labelValues = [DialFailure])
warn "failed to dial store peer, trying next"
lastError = StoreError(kind: ErrorCode.PEER_DIAL_FAILURE, address: $peer)
continue

12
logos_delivery/waku/waku_store/protocol.nim
View File

@ -45,7 +45,7 @@ proc handleQueryRequest(
let req = StoreQueryRequest.decode(raw_request).valueOr:
error "failed to decode rpc", peerId = requestor, error = $error
waku_store_errors.inc(labelValues = [DecodeRpcFailure])
logos_delivery_store_errors.inc(labelValues = [DecodeRpcFailure])
res.statusCode = uint32(ErrorCode.BAD_REQUEST)
res.statusDesc = "decoding rpc failed: " & $error
@ -56,7 +56,7 @@ proc handleQueryRequest(
info "received store query request",
peerId = requestor, requestId = requestId, request = req
waku_store_queries.inc()
logos_delivery_store_queries.inc()
let queryResult = await self.requestHandler(req)
@ -106,7 +106,7 @@ proc initProtocolHandler(self: WakuStore) =
error "Connection read error", error = error.msg
return
waku_service_network_bytes.inc(
logos_delivery_service_network_bytes.inc(
amount = reqBuf.len().int64, labelValues = [WakuStoreCodec, "in"]
)
@ -122,7 +122,7 @@ proc initProtocolHandler(self: WakuStore) =
return
queryDuration = getTime().toUnixFloat() - queryStartTime
waku_store_time_seconds.set(queryDuration, ["query-db-time"])
logos_delivery_store_time_seconds.set(queryDuration, ["query-db-time"])
successfulQuery = true
do:
info "store query request rejected due rate limit exceeded",
@ -140,13 +140,13 @@ proc initProtocolHandler(self: WakuStore) =
if successfulQuery:
let writeDuration = getTime().toUnixFloat() - writeRespStartTime
waku_store_time_seconds.set(writeDuration, ["send-store-resp-time"])
logos_delivery_store_time_seconds.set(writeDuration, ["send-store-resp-time"])
info "after sending response",
requestId = resBuf.requestId,
queryDurationSecs = queryDuration,
writeStreamDurationSecs = writeDuration
waku_service_network_bytes.inc(
logos_delivery_service_network_bytes.inc(
amount = resBuf.resp.len().int64, labelValues = [WakuStoreCodec, "out"]
)

11
logos_delivery/waku/waku_store/protocol_metrics.nim
View File

@ -2,22 +2,23 @@
import metrics
declarePublicCounter waku_store_errors, "number of store protocol errors", ["type"]
declarePublicCounter waku_store_queries, "number of store queries received"
declarePublicCounter logos_delivery_store_errors,
"number of store protocol errors", ["type"]
declarePublicCounter logos_delivery_store_queries, "number of store queries received"
## "query-db-time" phase considers the time when node performs the query to the database.
## "send-store-resp-time" phase is the time when node writes the store response to the store-client.
declarePublicGauge waku_store_time_seconds,
declarePublicGauge logos_delivery_store_time_seconds,
"Time in seconds spent by each store phase", labels = ["phase"]
declarePublicGauge(
waku_relay_fleet_store_msg_size_bytes,
logos_delivery_store_fleet_msg_size_bytes,
"Total size of messages stored by fleet store nodes per shard",
labels = ["shard"],
)
declarePublicGauge(
waku_relay_fleet_store_msg_count,
logos_delivery_store_fleet_msg_count,
"Number of messages stored by fleet store nodes per shard",
labels = ["shard"],
)

10
logos_delivery/waku/waku_store_sync/protocols_metrics.nim
View File

@ -6,19 +6,19 @@ const
Receiving* = "receive"
Sending* = "sent"
declarePublicHistogram reconciliation_roundtrips,
declarePublicHistogram logos_delivery_reconciliation_roundtrips,
"the nubmer of roundtrips for each reconciliation",
buckets = [1.0, 2.0, 3.0, 5.0, 8.0, 13.0, Inf]
declarePublicHistogram reconciliation_differences,
declarePublicHistogram logos_delivery_reconciliation_differences,
"the nubmer of differences for each reconciliation",
buckets = [0.0, 10.0, 50.0, 100.0, 500.0, 1000.0, 5000.0, Inf]
declarePublicCounter total_bytes_exchanged,
declarePublicCounter logos_delivery_total_bytes_exchanged,
"the number of bytes sent and received by the protocols", ["protocol", "direction"]
declarePublicCounter total_transfer_messages_exchanged,
declarePublicCounter logos_delivery_total_transfer_messages_exchanged,
"the number of messages sent and received by the transfer protocol", ["direction"]
declarePublicGauge total_messages_cached,
declarePublicGauge logos_delivery_total_messages_cached,
"the number of messages cached by the node after prunning"

18
logos_delivery/waku/waku_store_sync/reconciliation.nim
View File

@ -177,7 +177,9 @@ proc processRequest(
await conn.close()
return err("remote " & $conn.peerId & " connection read error: " & error.msg)
total_bytes_exchanged.inc(buffer.len, labelValues = [Reconciliation, Receiving])
logos_delivery_total_bytes_exchanged.inc(
buffer.len, labelValues = [Reconciliation, Receiving]
)
let recvPayload = RangesData.deltaDecode(buffer).valueOr:
await conn.close()
@ -226,7 +228,9 @@ proc processRequest(
rawPayload = sendPayload.deltaEncode()
total_bytes_exchanged.inc(rawPayload.len, labelValues = [Reconciliation, Sending])
logos_delivery_total_bytes_exchanged.inc(
rawPayload.len, labelValues = [Reconciliation, Sending]
)
let writeRes = catch:
await conn.writeLP(rawPayload)
@ -248,8 +252,8 @@ proc processRequest(
# Signal to transfer protocol that this reconciliation is done
await self.localWantsTx.addLast(conn.peerId)
reconciliation_roundtrips.observe(roundTrips)
reconciliation_differences.observe(diffs)
logos_delivery_reconciliation_roundtrips.observe(roundTrips)
logos_delivery_reconciliation_differences.observe(diffs)
await conn.close()
@ -281,7 +285,9 @@ proc initiate(
let sendPayload = initPayload.deltaEncode()
total_bytes_exchanged.inc(sendPayload.len, labelValues = [Reconciliation, Sending])
logos_delivery_total_bytes_exchanged.inc(
sendPayload.len, labelValues = [Reconciliation, Sending]
)
let writeRes = catch:
await connection.writeLP(sendPayload)
@ -456,7 +462,7 @@ proc periodicPrune(self: SyncReconciliation) {.async.} =
let count = self.storage.prune(time)
total_messages_cached.set(self.storage.length())
logos_delivery_total_messages_cached.set(self.storage.length())
info "periodic prune done", elements_pruned = count

12
logos_delivery/waku/waku_store_sync/transfer.nim
View File

@ -53,7 +53,9 @@ proc sendMessage(
): Future[Result[void, string]] {.async.} =
let rawPayload = payload.encode().buffer
total_bytes_exchanged.inc(rawPayload.len, labelValues = [Transfer, Sending])
logos_delivery_total_bytes_exchanged.inc(
rawPayload.len, labelValues = [Transfer, Sending]
)
let writeRes = catch:
await conn.writeLP(rawPayload)
@ -61,7 +63,7 @@ proc sendMessage(
writeRes.isOkOr:
return err("remote [" & $conn.peerId & "] connection write error: " & error.msg)
total_transfer_messages_exchanged.inc(labelValues = [Sending])
logos_delivery_total_transfer_messages_exchanged.inc(labelValues = [Sending])
return ok()
@ -156,13 +158,15 @@ proc initProtocolHandler(self: SyncTransfer) =
# connection closed normally
break
total_bytes_exchanged.inc(buffer.len, labelValues = [Transfer, Receiving])
logos_delivery_total_bytes_exchanged.inc(
buffer.len, labelValues = [Transfer, Receiving]
)
let payload = WakuMessageAndTopic.decode(buffer).valueOr:
error "decoding error", error = $error
continue
total_transfer_messages_exchanged.inc(labelValues = [Receiving])
logos_delivery_total_transfer_messages_exchanged.inc(labelValues = [Receiving])
let msg = payload.message
let pubsub = payload.pubsub

130
metrics/waku-fleet-dashboard.json
View File

@ -361,7 +361,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"format": "table",
"instant": true,
"legendFormat": "__auto",
@ -880,7 +880,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "avg by(topic, type)(rate(waku_relay_network_bytes_total{direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "avg by(topic, type)((rate(waku_relay_network_bytes_total{direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_relay_network_bytes_total{direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])))",
"instant": false,
"legendFormat": "{{topic}} - {{type}}",
"range": true,
@ -981,7 +981,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "avg by(topic, type)(rate(waku_relay_network_bytes_total{direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "avg by(topic, type)((rate(waku_relay_network_bytes_total{direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_relay_network_bytes_total{direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])))",
"legendFormat": "{{topic}}",
"range": true,
"refId": "A"
@ -1078,7 +1078,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "avg by (protocol)(waku_connected_peers{direction=\"In\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"expr": "avg by (protocol)((waku_connected_peers{direction=\"In\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_connected_peers{direction=\"In\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}))",
"instant": false,
"legendFormat": "{{protocol}}",
"range": true,
@ -1176,7 +1176,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "avg by (protocol)(waku_connected_peers{direction=\"Out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"expr": "avg by (protocol)((waku_connected_peers{direction=\"Out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_connected_peers{direction=\"Out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}))",
"instant": false,
"legendFormat": "{{protocol}}",
"range": true,
@ -1277,7 +1277,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "(increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))",
"expr": "((increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])))",
"interval": "",
"legendFormat": "{{type}}: {{instance}}",
"range": true,
@ -1472,7 +1472,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "count(count by (contentTopic)(waku_node_messages_total))",
"expr": "count(count by (contentTopic)((waku_node_messages_total or logos_delivery_node_messages_total)))",
"interval": "",
"legendFormat": "content topics",
"refId": "A"
@ -1569,7 +1569,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by (type)(increase(waku_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "sum by (type)((increase(waku_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))",
"interval": "",
"legendFormat": "peer {{type}}",
"range": true,
@ -1582,7 +1582,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by (type)(increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "sum by (type)((increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))",
"hide": false,
"interval": "",
"legendFormat": "store {{type}}",
@ -1596,7 +1596,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by (type)(increase(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "sum by (type)((increase(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))",
"hide": false,
"interval": "",
"legendFormat": "node {{type}}",
@ -1698,7 +1698,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "waku_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}:{{version}}",
"range": true,
@ -1906,7 +1906,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{type}}: {{instance}}",
"refId": "A"
@ -2325,7 +2325,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -3400,7 +3400,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_total_generated_proofs",
"expr": "(waku_rln_total_generated_proofs or logos_delivery_rln_total_generated_proofs)",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -3496,7 +3496,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_proof_verification_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_verification_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_rln_proof_verification_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -3593,7 +3593,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_remaining_proofs_per_epoch",
"expr": "(waku_rln_remaining_proofs_per_epoch or logos_delivery_rln_remaining_proofs_per_epoch)",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -3690,7 +3690,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -3786,7 +3786,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -3882,7 +3882,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "changes(waku_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[14m])",
"expr": "(changes(waku_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[14m]) or changes(logos_delivery_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[14m]))",
"hide": false,
"legendFormat": "Proofs generated: {{instance}}",
"range": true,
@ -3979,7 +3979,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"legendFormat": "{{type}} : {{instance}}",
"range": true,
"refId": "A"
@ -3990,7 +3990,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"hide": false,
"legendFormat": "spam : {{instance}}",
"range": true,
@ -4103,7 +4103,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "waku_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -4215,7 +4215,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"hide": false,
"legendFormat": "{{type}}: {{instance}}",
"range": true,
@ -4314,7 +4314,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "increase(waku_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])",
"expr": "(increase(waku_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -4413,7 +4413,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "sum by (type)(increase(waku_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]))",
"expr": "sum by (type)((increase(waku_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m])))",
"hide": false,
"interval": "",
"legendFormat": "{{type}}",
@ -4427,7 +4427,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "sum by (type)(increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]))",
"expr": "sum by (type)((increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m])))",
"hide": false,
"interval": "",
"legendFormat": "{{type}}",
@ -4534,7 +4534,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"sendQuery\"} and deriv(query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"sendQuery\"}[1m]) != 0",
"expr": "(query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"sendQuery\"} or logos_delivery_query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"sendQuery\"}) and (deriv(query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"sendQuery\"}[1m]) or deriv(logos_delivery_query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"sendQuery\"}[1m])) != 0",
"interval": "",
"legendFormat": "{{query}}",
"range": true,
@ -4639,7 +4639,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"waitFinish\"} and deriv(query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"waitFinish\"}[45s]) != 0",
"expr": "(query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"waitFinish\"} or logos_delivery_query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"waitFinish\"}) and (deriv(query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"waitFinish\"}[45s]) or deriv(logos_delivery_query_time_secs{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", phase=\"waitFinish\"}[45s])) != 0",
"hide": false,
"interval": "",
"legendFormat": "{{query}}",
@ -4850,7 +4850,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} and deriv(waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[45s]) != 0",
"expr": "(waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}) and (deriv(waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[45s]) or deriv(logos_delivery_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[45s])) != 0",
"interval": "",
"legendFormat": "{{phase}}",
"range": true,
@ -5058,7 +5058,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "builder",
"expr": "sum by(instance) (waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} unless deriv(waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[45s]) == 0)\n",
"expr": "sum by(instance) ((waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}) unless (deriv(waku_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[45s]) or deriv(logos_delivery_store_time_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[45s])) == 0)\n",
"hide": false,
"legendFormat": "{{instance}}",
"range": true,
@ -5157,7 +5157,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "rate(query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query!=\"InsertRow\"}[1m]) and rate(query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query!=\"InsertRowMessagesLookup\"}[1m]) ",
"expr": "(rate(query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query!=\"InsertRow\"}[1m]) or rate(logos_delivery_query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query!=\"InsertRow\"}[1m])) and (rate(query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query!=\"InsertRowMessagesLookup\"}[1m]) or rate(logos_delivery_query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query!=\"InsertRowMessagesLookup\"}[1m])) ",
"interval": "",
"legendFormat": "{{query}}",
"range": true,
@ -5281,7 +5281,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "rate(query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query=\"InsertRow\"}[5m])",
"expr": "(rate(query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query=\"InsertRow\"}[5m]) or rate(logos_delivery_query_count_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\", query=\"InsertRow\"}[5m]))",
"interval": "",
"legendFormat": "{{query}}",
"range": true,
@ -5367,7 +5367,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "sum(increase(waku_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"hide": false,
"legendFormat": "{{le}}",
@ -5467,7 +5467,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(increase(waku_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"hide": false,
"instant": false,
@ -5605,7 +5605,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{service=~\"/vac/waku/store/2.*\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{service=~\"/vac/waku/store/2.*\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{service=~\"/vac/waku/store/2.*\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -5707,7 +5707,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{service=\"/vac/waku/store-query/3.0.0\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{service=\"/vac/waku/store-query/3.0.0\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{service=\"/vac/waku/store-query/3.0.0\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -5811,7 +5811,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{service=~\"/vac/waku/store/2.*\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{service=~\"/vac/waku/store/2.*\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{service=~\"/vac/waku/store/2.*\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -5915,7 +5915,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{service=\"/vac/waku/store-query/3.0.0\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{service=\"/vac/waku/store-query/3.0.0\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{service=\"/vac/waku/store-query/3.0.0\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -6019,7 +6019,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_requests_total{service =~\"/vac/waku/store/2.*\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_requests_total{service =~\"/vac/waku/store/2.*\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_requests_total{service =~\"/vac/waku/store/2.*\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}} - {{state}}",
"range": true,
"refId": "A"
@ -6123,7 +6123,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_requests_total{service =~\"/vac/waku/store-query/3.*\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_requests_total{service =~\"/vac/waku/store-query/3.*\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_requests_total{service =~\"/vac/waku/store-query/3.*\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}} - {{state}}",
"range": true,
"refId": "A"
@ -6236,7 +6236,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sum by(direction, protocol) (total_bytes_exchanged_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"expr": "sum by(direction, protocol) ((total_bytes_exchanged_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_total_bytes_exchanged_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}))",
"fullMetaSearch": false,
"includeNullMetadata": false,
"legendFormat": "__auto",
@ -6334,7 +6334,7 @@
},
"disableTextWrap": false,
"editorMode": "code",
"expr": "sum by(direction) (total_transfer_messages_exchanged_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"expr": "sum by(direction) ((total_transfer_messages_exchanged_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_total_transfer_messages_exchanged_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}))",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "__auto",
@ -6411,7 +6411,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "sum by(le) (reconciliation_roundtrips_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"expr": "sum by(le) ((reconciliation_roundtrips_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_reconciliation_roundtrips_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}))",
"format": "heatmap",
"fullMetaSearch": false,
"hide": false,
@ -6485,7 +6485,7 @@
{
"disableTextWrap": false,
"editorMode": "builder",
"expr": "sum by(le) (reconciliation_differences_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"expr": "sum by(le) ((reconciliation_differences_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_reconciliation_differences_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}))",
"format": "heatmap",
"fullMetaSearch": false,
"includeNullMetadata": true,
@ -6581,7 +6581,7 @@
"disableTextWrap": false,
"editorMode": "code",
"exemplar": false,
"expr": "total_messages_cached{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(total_messages_cached{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_total_messages_cached{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"format": "time_series",
"fullMetaSearch": false,
"includeNullMetadata": true,
@ -6695,7 +6695,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "rate(waku_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{type}} : {{instance}}",
"range": true,
"refId": "A"
@ -6791,7 +6791,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -6887,7 +6887,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "rate(waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{type}} : {{instance}}",
"range": true,
"refId": "A"
@ -6967,7 +6967,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "sum(increase(waku_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"interval": "",
"legendFormat": "{{le}}",
@ -7049,7 +7049,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "sum(increase(waku_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"legendFormat": "{{le}}",
"range": true,
@ -7153,7 +7153,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_requests_total{service = \"/vac/waku/filter-subscribe/2.0.0-beta1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_requests_total{service = \"/vac/waku/filter-subscribe/2.0.0-beta1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_requests_total{service = \"/vac/waku/filter-subscribe/2.0.0-beta1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}} - {{state}}",
"range": true,
"refId": "A"
@ -7254,7 +7254,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{service=\"/vac/waku/filter-push/2.0.0-beta1\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{service=\"/vac/waku/filter-push/2.0.0-beta1\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{service=\"/vac/waku/filter-push/2.0.0-beta1\", direction=\"out\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -7364,7 +7364,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -7460,7 +7460,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{type}}: {[instance}}",
"refId": "A"
@ -7563,7 +7563,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_requests_total{service = \"/vac/waku/lightpush/2.0.0-beta1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_requests_total{service = \"/vac/waku/lightpush/2.0.0-beta1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_requests_total{service = \"/vac/waku/lightpush/2.0.0-beta1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}} . {{state}}",
"range": true,
"refId": "A"
@ -7664,7 +7664,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{service=\"/vac/waku/lightpush/2.0.0-beta1\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{service=\"/vac/waku/lightpush/2.0.0-beta1\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{service=\"/vac/waku/lightpush/2.0.0-beta1\", direction=\"in\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -7782,7 +7782,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_requests_total{service = \"/vac/waku/peer-exchange/2.0.0-alpha1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_service_requests_total{service = \"/vac/waku/peer-exchange/2.0.0-alpha1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_service_requests_total{service = \"/vac/waku/peer-exchange/2.0.0-alpha1\", instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}} . {{state}}",
"range": true,
"refId": "A"
@ -7885,7 +7885,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_px_peers_sent_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_px_peers_sent_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_px_peers_sent_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -7961,7 +7961,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "builder",
"expr": "sum by(instance, type) (waku_px_errors_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"expr": "sum by(instance, type) ((waku_px_errors_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_px_errors_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}))",
"legendFormat": "{{instance}} - {{type}}",
"range": true,
"refId": "A"
@ -8064,7 +8064,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_px_errors_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval])",
"expr": "(rate(waku_px_errors_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]) or rate(logos_delivery_px_errors_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[$__rate_interval]))",
"legendFormat": "{{instance}} - {{type}}",
"range": true,
"refId": "A"
@ -8543,7 +8543,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "(increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))/60",
"expr": "((increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])))/60",
"format": "heatmap",
"instant": false,
"legendFormat": "{{fleet}}_{{datacenter}}",
@ -8639,7 +8639,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}/waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})/(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "heatmap",
"instant": false,
"legendFormat": "{{fleet}}_{{datacenter}}",
@ -8713,7 +8713,7 @@
},
"editorMode": "builder",
"exemplar": false,
"expr": "rate(waku_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h])/scalar(rate(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]))*100",
"expr": "(rate(waku_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]) or rate(logos_delivery_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]))/scalar((rate(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]) or rate(logos_delivery_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h])))*100",
"format": "heatmap",
"instant": false,
"legendFormat": "{{le}}",
@ -8784,7 +8784,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "heatmap",
"instant": false,
"legendFormat": "Direction:{{direction}} {{protocol}}",
@ -8855,7 +8855,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "heatmap",
"instant": false,
"legendFormat": "Direction:{{direction}} {{protocol}}",

68
metrics/waku-network-monitor-dashboard.json
View File

@ -167,7 +167,7 @@
},
"editorMode": "builder",
"exemplar": false,
"expr": "waku_version{instance=\"nwaku:8003\"}",
"expr": "(waku_version{instance=\"nwaku:8003\"} or logos_delivery_version{instance=\"nwaku:8003\"})",
"format": "table",
"instant": true,
"legendFormat": "__auto",
@ -610,7 +610,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "waku_connected_peers",
"expr": "(waku_connected_peers or logos_delivery_connected_peers)",
"legendFormat": "{{direction}}_{{protocol}}",
"range": true,
"refId": "A"
@ -773,7 +773,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_discv5_discovered_per_shard",
"expr": "(waku_discv5_discovered_per_shard or logos_delivery_discv5_discovered_per_shard)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -842,7 +842,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_connected_peers_per_shard",
"expr": "(waku_connected_peers_per_shard or logos_delivery_connected_peers_per_shard)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -911,7 +911,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_connected_peers_per_agent",
"expr": "(waku_connected_peers_per_agent or logos_delivery_connected_peers_per_agent)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "{{agent}}",
@ -1022,7 +1022,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_histogram_message_size_sum/waku_histogram_message_size_count",
"expr": "(waku_histogram_message_size_sum or logos_delivery_histogram_message_size_sum)/(waku_histogram_message_size_count or logos_delivery_histogram_message_size_count)",
"format": "heatmap",
"instant": false,
"legendFormat": "{{instance}}",
@ -1095,7 +1095,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "rate(waku_histogram_message_size_bucket[1h])/scalar(rate(waku_histogram_message_size_count[1h]))*100",
"expr": "(rate(waku_histogram_message_size_bucket[1h]) or rate(logos_delivery_histogram_message_size_bucket[1h]))/scalar((rate(waku_histogram_message_size_count[1h]) or rate(logos_delivery_histogram_message_size_count[1h])))*100",
"format": "heatmap",
"instant": true,
"legendFormat": "__auto",
@ -1166,7 +1166,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_histogram_message_size_sum[1h])/rate(waku_histogram_message_size_count[1h])",
"expr": "(rate(waku_histogram_message_size_sum[1h]) or rate(logos_delivery_histogram_message_size_sum[1h]))/(rate(waku_histogram_message_size_count[1h]) or rate(logos_delivery_histogram_message_size_count[1h]))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -1231,7 +1231,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "histogram_quantile(0.75, rate(waku_histogram_message_size_bucket[1h]))",
"expr": "histogram_quantile(0.75, (rate(waku_histogram_message_size_bucket[1h]) or rate(logos_delivery_histogram_message_size_bucket[1h])))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -1300,7 +1300,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, rate(waku_histogram_message_size_bucket[1h]))",
"expr": "histogram_quantile(0.99, (rate(waku_histogram_message_size_bucket[1h]) or rate(logos_delivery_histogram_message_size_bucket[1h])))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -1560,7 +1560,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "waku_peer_store_size",
"expr": "(waku_peer_store_size or logos_delivery_peer_store_size)",
"legendFormat": "__auto",
"range": true,
"refId": "A"
@ -1658,7 +1658,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_requests_total[$__rate_interval])",
"expr": "(rate(waku_service_requests_total[$__rate_interval]) or rate(logos_delivery_service_requests_total[$__rate_interval]))",
"legendFormat": "{{proto}}/{{state}}",
"range": true,
"refId": "A"
@ -1757,7 +1757,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "rate(waku_relay_network_bytes_total{direction=\"in\"}[$__rate_interval])*8",
"expr": "(rate(waku_relay_network_bytes_total{direction=\"in\"}[$__rate_interval]) or rate(logos_delivery_relay_network_bytes_total{direction=\"in\"}[$__rate_interval]))*8",
"legendFormat": "{{topic}}/{{type}}",
"range": true,
"refId": "A"
@ -1855,7 +1855,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_relay_network_bytes_total{direction=\"out\"}[$__rate_interval])*8",
"expr": "(rate(waku_relay_network_bytes_total{direction=\"out\"}[$__rate_interval]) or rate(logos_delivery_relay_network_bytes_total{direction=\"out\"}[$__rate_interval]))*8",
"legendFormat": "{{topic}}",
"range": true,
"refId": "A"
@ -1953,7 +1953,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{direction=\"in\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{direction=\"in\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{direction=\"in\"}[$__rate_interval]))",
"legendFormat": "{{service}}",
"range": true,
"refId": "A"
@ -2051,7 +2051,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "rate(waku_service_network_bytes_total{direction=\"out\"}[$__rate_interval])",
"expr": "(rate(waku_service_network_bytes_total{direction=\"out\"}[$__rate_interval]) or rate(logos_delivery_service_network_bytes_total{direction=\"out\"}[$__rate_interval]))",
"legendFormat": "{{service}}",
"range": true,
"refId": "A"
@ -2117,7 +2117,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_relay_total_msg_bytes_per_shard",
"expr": "(waku_relay_total_msg_bytes_per_shard or logos_delivery_relay_total_msg_bytes_per_shard)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -2186,7 +2186,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_relay_max_msg_bytes_per_shard",
"expr": "(waku_relay_max_msg_bytes_per_shard or logos_delivery_relay_max_msg_bytes_per_shard)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -2255,7 +2255,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_relay_avg_msg_bytes_per_shard",
"expr": "(waku_relay_avg_msg_bytes_per_shard or logos_delivery_relay_avg_msg_bytes_per_shard)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -2324,7 +2324,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_archive_messages_per_shard",
"expr": "(waku_archive_messages_per_shard or logos_delivery_archive_messages_per_shard)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -2393,7 +2393,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_relay_fleet_store_msg_size_bytes",
"expr": "(waku_relay_fleet_store_msg_size_bytes or logos_delivery_store_fleet_msg_size_bytes)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -2462,7 +2462,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_relay_fleet_store_msg_count",
"expr": "(waku_relay_fleet_store_msg_count or logos_delivery_store_fleet_msg_count)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"legendFormat": "shard-{{shard}}",
@ -2980,7 +2980,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_membership_insertion_duration_seconds",
"expr": "(waku_rln_membership_insertion_duration_seconds or logos_delivery_rln_membership_insertion_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3082,7 +3082,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_membership_credentials_import_duration_seconds",
"expr": "(waku_rln_membership_credentials_import_duration_seconds or logos_delivery_rln_membership_credentials_import_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3184,7 +3184,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_number_registered_memberships",
"expr": "(waku_rln_number_registered_memberships or logos_delivery_rln_number_registered_memberships)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3286,7 +3286,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_proof_generation_duration_seconds",
"expr": "(waku_rln_proof_generation_duration_seconds or logos_delivery_rln_proof_generation_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3388,7 +3388,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_proof_verification_duration_seconds",
"expr": "(waku_rln_proof_verification_duration_seconds or logos_delivery_rln_proof_verification_duration_seconds)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3490,7 +3490,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_remaining_proofs_per_epoch",
"expr": "(waku_rln_remaining_proofs_per_epoch or logos_delivery_rln_remaining_proofs_per_epoch)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3592,7 +3592,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_total_generated_proofs",
"expr": "(waku_rln_total_generated_proofs or logos_delivery_rln_total_generated_proofs)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3694,7 +3694,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_proof_verification_total_total",
"expr": "(waku_rln_proof_verification_total_total or logos_delivery_rln_proof_verification_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3796,7 +3796,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_valid_messages_total_total",
"expr": "(waku_rln_valid_messages_total_total or logos_delivery_rln_valid_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -3898,7 +3898,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_invalid_messages_total_total",
"expr": "(waku_rln_invalid_messages_total_total or logos_delivery_rln_invalid_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -4000,7 +4000,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "waku_rln_spam_messages_total_total",
"expr": "(waku_rln_spam_messages_total_total or logos_delivery_rln_spam_messages_total_total)",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -5356,7 +5356,7 @@
"uid": "PBFA97CFB590B2093"
},
"editorMode": "code",
"expr": "(increase(waku_node_messages_total[1m]))/60",
"expr": "((increase(waku_node_messages_total[1m]) or increase(logos_delivery_node_messages_total[1m])))/60",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"

32
metrics/waku-network-monitor-discovery.json
View File

@ -108,7 +108,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "networkmonitor_peer_count{connected=\"false\"}",
"expr": "(networkmonitor_peer_count{connected=\"false\"} or logos_delivery_networkmonitor_peer_count{connected=\"false\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": false,
@ -124,7 +124,7 @@
},
"disableTextWrap": false,
"editorMode": "builder",
"expr": "networkmonitor_peer_count{connected=\"true\"}",
"expr": "(networkmonitor_peer_count{connected=\"true\"} or logos_delivery_networkmonitor_peer_count{connected=\"true\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -199,7 +199,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/filter-subscribe/2.0.0-beta1\"}",
"expr": "(networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/filter-subscribe/2.0.0-beta1\"} or logos_delivery_networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/filter-subscribe/2.0.0-beta1\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": true,
@ -216,7 +216,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/lightpush/2.0.0-beta1\"}",
"expr": "(networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/lightpush/2.0.0-beta1\"} or logos_delivery_networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/lightpush/2.0.0-beta1\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -234,7 +234,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/peer-exchange/2.0.0-alpha1\"}",
"expr": "(networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/peer-exchange/2.0.0-alpha1\"} or logos_delivery_networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/peer-exchange/2.0.0-alpha1\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -252,7 +252,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/relay/2.0.0\"}",
"expr": "(networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/relay/2.0.0\"} or logos_delivery_networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/relay/2.0.0\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -270,7 +270,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/store-query/3.0.0\"}",
"expr": "(networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/store-query/3.0.0\"} or logos_delivery_networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/store-query/3.0.0\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -288,7 +288,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/store/2.0.0-beta4\"}",
"expr": "(networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/store/2.0.0-beta4\"} or logos_delivery_networkmonitor_peer_type_as_per_protocol{protocols=\"/vac/waku/store/2.0.0-beta4\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -573,7 +573,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_cluster_as_per_enr{cluster=\"1\"}",
"expr": "(networkmonitor_peer_cluster_as_per_enr{cluster=\"1\"} or logos_delivery_networkmonitor_peer_cluster_as_per_enr{cluster=\"1\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": true,
@ -590,7 +590,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_cluster_as_per_enr{cluster=\"16\"}",
"expr": "(networkmonitor_peer_cluster_as_per_enr{cluster=\"16\"} or logos_delivery_networkmonitor_peer_cluster_as_per_enr{cluster=\"16\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -608,7 +608,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_cluster_as_per_enr{cluster=\"0\"}",
"expr": "(networkmonitor_peer_cluster_as_per_enr{cluster=\"0\"} or logos_delivery_networkmonitor_peer_cluster_as_per_enr{cluster=\"0\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -626,7 +626,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_cluster_as_per_enr{cluster!~\"0|16|1\"}",
"expr": "(networkmonitor_peer_cluster_as_per_enr{cluster!~\"0|16|1\"} or logos_delivery_networkmonitor_peer_cluster_as_per_enr{cluster!~\"0|16|1\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -701,7 +701,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_enr{capability=\"Filter\"}",
"expr": "(networkmonitor_peer_type_as_per_enr{capability=\"Filter\"} or logos_delivery_networkmonitor_peer_type_as_per_enr{capability=\"Filter\"})",
"fullMetaSearch": false,
"includeNullMetadata": true,
"instant": true,
@ -718,7 +718,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_enr{capability=\"Lightpush\"}",
"expr": "(networkmonitor_peer_type_as_per_enr{capability=\"Lightpush\"} or logos_delivery_networkmonitor_peer_type_as_per_enr{capability=\"Lightpush\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -736,7 +736,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_enr{capability=\"Relay\"}",
"expr": "(networkmonitor_peer_type_as_per_enr{capability=\"Relay\"} or logos_delivery_networkmonitor_peer_type_as_per_enr{capability=\"Relay\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,
@ -754,7 +754,7 @@
"disableTextWrap": false,
"editorMode": "builder",
"exemplar": false,
"expr": "networkmonitor_peer_type_as_per_enr{capability=\"Store\"}",
"expr": "(networkmonitor_peer_type_as_per_enr{capability=\"Store\"} or logos_delivery_networkmonitor_peer_type_as_per_enr{capability=\"Store\"})",
"fullMetaSearch": false,
"hide": false,
"includeNullMetadata": true,

18
metrics/waku-rln-relay-fleet-dashboard.json
View File

@ -111,7 +111,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_membership_insertion_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_membership_insertion_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_membership_insertion_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -203,7 +203,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -295,7 +295,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_proof_verification_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_verification_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_verification_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -388,7 +388,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -481,7 +481,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -665,7 +665,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_membership_credentials_import_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_membership_credentials_import_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_membership_credentials_import_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -757,7 +757,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_valid_messages_total_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_valid_messages_total_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_valid_messages_total_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -849,7 +849,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_generation_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -941,7 +941,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_valid_messages_total_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_valid_messages_total_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_valid_messages_total_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "time_series",
"interval": "",
"intervalFactor": 1,

18
metrics/waku-rln-relay-single-node-dashboard.json
View File

@ -101,7 +101,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_membership_insertion_duration_seconds{}",
"expr": "(waku_rln_membership_insertion_duration_seconds{} or logos_delivery_rln_membership_insertion_duration_seconds{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -186,7 +186,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_messages_total_total{}",
"expr": "(waku_rln_messages_total_total{} or logos_delivery_rln_messages_total_total{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -271,7 +271,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_proof_verification_total_total{}",
"expr": "(waku_rln_proof_verification_total_total{} or logos_delivery_rln_proof_verification_total_total{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -441,7 +441,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_spam_messages_total_total{}",
"expr": "(waku_rln_spam_messages_total_total{} or logos_delivery_rln_spam_messages_total_total{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -526,7 +526,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_valid_messages_total_sum{}",
"expr": "(waku_rln_valid_messages_total_sum{} or logos_delivery_rln_valid_messages_total_sum{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -611,7 +611,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_membership_credentials_import_duration_seconds{}",
"expr": "(waku_rln_membership_credentials_import_duration_seconds{} or logos_delivery_rln_membership_credentials_import_duration_seconds{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -696,7 +696,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_valid_messages_total_bucket{}",
"expr": "(waku_rln_valid_messages_total_bucket{} or logos_delivery_rln_valid_messages_total_bucket{})",
"format": "time_series",
"interval": "",
"intervalFactor": 1,
@ -783,7 +783,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_proof_generation_duration_seconds{}",
"expr": "(waku_rln_proof_generation_duration_seconds{} or logos_delivery_rln_proof_generation_duration_seconds{})",
"interval": "",
"legendFormat": "",
"refId": "A"
@ -953,7 +953,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_rln_proof_verification_duration_seconds{}",
"expr": "(waku_rln_proof_verification_duration_seconds{} or logos_delivery_rln_proof_verification_duration_seconds{})",
"interval": "",
"legendFormat": "",
"refId": "A"

72
metrics/waku-single-node-dashboard.json
View File

@ -362,7 +362,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_version{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"format": "table",
"instant": true,
"legendFormat": "__auto",
@ -879,7 +879,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "(increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))",
"expr": "((increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])))",
"interval": "",
"legendFormat": "{{type}}: {{instance}}",
"range": true,
@ -978,7 +978,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by (type)(increase(waku_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "sum by (type)((increase(waku_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_peers_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))",
"interval": "",
"legendFormat": "peer {{type}}",
"range": true,
@ -991,7 +991,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by (type)(increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "sum by (type)((increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))",
"hide": false,
"interval": "",
"legendFormat": "store {{type}}",
@ -1005,7 +1005,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by (type)(increase(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))",
"expr": "sum by (type)((increase(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])))",
"hide": false,
"interval": "",
"legendFormat": "node {{type}}",
@ -1104,7 +1104,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "count(count by (contentTopic)(waku_node_messages_total))",
"expr": "count(count by (contentTopic)((waku_node_messages_total or logos_delivery_node_messages_total)))",
"interval": "",
"legendFormat": "content topics",
"refId": "A"
@ -1205,7 +1205,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "waku_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_version{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}:{{version}}",
"range": true,
@ -1414,7 +1414,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_node_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{type}}: {{instance}}",
"refId": "A"
@ -1839,7 +1839,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_peer_store_size{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -2947,7 +2947,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "increase(waku_node_messages_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m])",
"expr": "(increase(waku_node_messages_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m]) or increase(logos_delivery_node_messages_total{instance=~\"bridge*[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[10m]))",
"hide": false,
"interval": "",
"legendFormat": "{{fleet}} : v2_messages",
@ -3168,7 +3168,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_store_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -3273,7 +3273,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_archive_messages{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"hide": false,
"legendFormat": "{{type}}: {{instance}}",
"range": true,
@ -3366,7 +3366,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "increase(waku_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])",
"expr": "(increase(waku_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_store_queries_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))",
"interval": "",
"legendFormat": "{{instance}}",
"range": true,
@ -3459,7 +3459,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "sum by (type)(increase(waku_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]))",
"expr": "sum by (type)((increase(waku_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_archive_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m])))",
"hide": false,
"interval": "",
"legendFormat": "{{type}}",
@ -3473,7 +3473,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "sum by (type)(increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]))",
"expr": "sum by (type)((increase(waku_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_store_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[1m])))",
"hide": false,
"interval": "",
"legendFormat": "{{type}}",
@ -3558,7 +3558,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "sum(increase(waku_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_archive_query_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"hide": false,
"legendFormat": "{{le}}",
@ -3657,7 +3657,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(increase(waku_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_archive_insert_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"hide": false,
"instant": false,
@ -3782,7 +3782,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_requests_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"legendFormat": "{{type}} : {{instance}}",
"range": true,
"refId": "A"
@ -3873,7 +3873,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_subscriptions{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
@ -3964,7 +3964,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"legendFormat": "{{type}} : {{instance}}",
"range": true,
"refId": "A"
@ -4043,7 +4043,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "sum(increase(waku_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_filter_request_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"interval": "",
"legendFormat": "{{le}}",
@ -4124,7 +4124,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "sum(increase(waku_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval])) by (le)",
"expr": "sum((increase(waku_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]) or increase(logos_delivery_filter_handle_message_duration_seconds_bucket{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}[$__rate_interval]))) by (le)",
"format": "heatmap",
"legendFormat": "{{le}}",
"range": true,
@ -4231,7 +4231,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_filter_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_filter_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"hide": false,
"interval": "",
"legendFormat": "{{instance}}",
@ -4325,7 +4325,7 @@
},
"editorMode": "code",
"exemplar": true,
"expr": "waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_filter_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{type}}: {{instance}}",
"range": true,
@ -4651,7 +4651,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_lightpush_peers{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{instance}}",
"refId": "A"
@ -4742,7 +4742,7 @@
"uid": "P6693426190CB2316"
},
"exemplar": true,
"expr": "waku_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"}",
"expr": "(waku_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"} or logos_delivery_lightpush_errors_total{instance=~\"[[host]].([[dc:pipe]]).*.([[fleet:pipe]])\"})",
"interval": "",
"legendFormat": "{{type}}: {[instance}}",
"refId": "A"
@ -4846,7 +4846,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "(increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]))/60",
"expr": "((increase(waku_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m]) or increase(logos_delivery_node_messages_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1m])))/60",
"format": "heatmap",
"instant": false,
"legendFormat": "{{fleet}}_{{datacenter}}",
@ -4938,7 +4938,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}/waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_histogram_message_size_sum{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})/(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "heatmap",
"instant": false,
"legendFormat": "{{fleet}}_{{datacenter}}",
@ -5002,7 +5002,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "rate(waku_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h])/scalar(rate(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]))*100",
"expr": "(rate(waku_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]) or rate(logos_delivery_histogram_message_size_bucket{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]))/scalar((rate(waku_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h]) or rate(logos_delivery_histogram_message_size_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}[1h])))*100",
"format": "heatmap",
"instant": false,
"legendFormat": "{{le}}",
@ -5070,7 +5070,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_connected_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "heatmap",
"instant": false,
"legendFormat": "Direction:{{direction}} {{protocol}}",
@ -5138,7 +5138,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "waku_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_streams_peers{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"format": "heatmap",
"instant": false,
"legendFormat": "Direction:{{direction}} {{protocol}}",
@ -5254,7 +5254,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_invalid_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{type}}_{{datacenter}}",
"range": true,
"refId": "A"
@ -5351,7 +5351,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_valid_messages_total_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_valid_messages_total_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_valid_messages_total_count{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{__name__}}_{{datacenter}}",
"range": true,
"refId": "A"
@ -5448,7 +5448,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_spam_messages_total_total{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{__name__}}_{{datacenter}}",
"range": true,
"refId": "A"
@ -5545,7 +5545,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_proof_verification_duration_seconds{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{__name__}}_{{datacenter}}",
"range": true,
"refId": "A"
@ -5642,7 +5642,7 @@
"uid": "P6693426190CB2316"
},
"editorMode": "code",
"expr": "waku_rln_number_registered_memberships{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"}",
"expr": "(waku_rln_number_registered_memberships{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"} or logos_delivery_rln_number_registered_memberships{instance=~\"[[host]].([[dc:pipe]]).([[fleet:pipe]])\"})",
"legendFormat": "{{__name__}}_{{datacenter}}",
"range": true,
"refId": "A"

234
tests-e2e/src/test_data.py
View File

@ -201,17 +201,17 @@ METRICS_WITH_INITIAL_VALUE_ZERO = [
"libp2p_total_dial_attempts_total",
"libp2p_successful_dials_total",
"libp2p_failed_dials_total",
"waku_rln_messages_total_total",
"waku_rln_spam_messages_total_total",
"waku_rln_proof_verification_total_total",
"waku_rln_number_registered_memberships",
"waku_rln_proof_verification_duration_seconds",
"waku_rln_proof_generation_duration_seconds",
"waku_rln_instance_creation_duration_seconds",
"waku_rln_membership_insertion_duration_seconds",
"waku_rln_membership_credentials_import_duration_seconds",
"waku_rln_remaining_proofs_per_epoch",
"waku_rln_total_generated_proofs",
"logos_delivery_rln_messages_total_total",
"logos_delivery_rln_spam_messages_total_total",
"logos_delivery_rln_proof_verification_total_total",
"logos_delivery_rln_number_registered_memberships",
"logos_delivery_rln_proof_verification_duration_seconds",
"logos_delivery_rln_proof_generation_duration_seconds",
"logos_delivery_rln_instance_creation_duration_seconds",
"logos_delivery_rln_membership_insertion_duration_seconds",
"logos_delivery_rln_membership_credentials_import_duration_seconds",
"logos_delivery_rln_remaining_proofs_per_epoch",
"logos_delivery_rln_total_generated_proofs",
"libp2p_pubsub_sig_verify_success_total",
"libp2p_pubsub_sig_verify_failure_total",
"libp2p_pubsub_disconnects_over_non_priority_queue_limit_total",
@ -238,119 +238,119 @@ METRICS_WITH_INITIAL_VALUE_ZERO = [
"libp2p_rendezvous_discover_total",
"libp2p_rendezvous_registered",
"libp2p_rendezvous_namespaces",
"waku_peer_store_size",
"waku_total_unique_peers",
"logos_delivery_peer_store_size",
"logos_delivery_total_unique_peers",
"presto_server_missing_requests_count",
"presto_server_invalid_requests_count",
'waku_archive_messages{type="stored"}',
"waku_archive_insert_duration_seconds_sum",
"waku_archive_insert_duration_seconds_count",
'waku_archive_insert_duration_seconds_bucket{le="0.005"}',
'waku_archive_insert_duration_seconds_bucket{le="0.01"}',
'waku_archive_insert_duration_seconds_bucket{le="0.025"}',
'waku_archive_insert_duration_seconds_bucket{le="0.05"}',
'waku_archive_insert_duration_seconds_bucket{le="0.075"}',
'waku_archive_insert_duration_seconds_bucket{le="0.1"}',
'waku_archive_insert_duration_seconds_bucket{le="0.25"}',
'waku_archive_insert_duration_seconds_bucket{le="0.5"}',
'waku_archive_insert_duration_seconds_bucket{le="0.75"}',
'waku_archive_insert_duration_seconds_bucket{le="1.0"}',
'waku_archive_insert_duration_seconds_bucket{le="2.5"}',
'waku_archive_insert_duration_seconds_bucket{le="5.0"}',
'waku_archive_insert_duration_seconds_bucket{le="7.5"}',
'waku_archive_insert_duration_seconds_bucket{le="10.0"}',
'waku_archive_insert_duration_seconds_bucket{le="+Inf"}',
"waku_archive_query_duration_seconds_sum",
"waku_archive_query_duration_seconds_count",
'waku_archive_query_duration_seconds_bucket{le="0.005"}',
'waku_archive_query_duration_seconds_bucket{le="0.01"}',
'waku_archive_query_duration_seconds_bucket{le="0.025"}',
'waku_archive_query_duration_seconds_bucket{le="0.05"}',
'waku_archive_query_duration_seconds_bucket{le="0.075"}',
'waku_archive_query_duration_seconds_bucket{le="0.1"}',
'waku_archive_query_duration_seconds_bucket{le="0.25"}',
'waku_archive_query_duration_seconds_bucket{le="0.5"}',
'waku_archive_query_duration_seconds_bucket{le="0.75"}',
'waku_archive_query_duration_seconds_bucket{le="1.0"}',
'waku_archive_query_duration_seconds_bucket{le="2.5"}',
'waku_archive_query_duration_seconds_bucket{le="5.0"}',
'waku_archive_query_duration_seconds_bucket{le="7.5"}',
'waku_archive_query_duration_seconds_bucket{le="10.0"}',
'waku_archive_query_duration_seconds_bucket{le="+Inf"}',
"waku_filter_subscriptions",
"waku_filter_handle_message_duration_seconds_sum",
"waku_filter_handle_message_duration_seconds_count",
'waku_filter_handle_message_duration_seconds_bucket{le="0.005"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.01"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.025"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.05"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.075"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.1"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.25"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.5"}',
'waku_filter_handle_message_duration_seconds_bucket{le="0.75"}',
'waku_filter_handle_message_duration_seconds_bucket{le="1.0"}',
'waku_filter_handle_message_duration_seconds_bucket{le="2.5"}',
'waku_filter_handle_message_duration_seconds_bucket{le="5.0"}',
'waku_filter_handle_message_duration_seconds_bucket{le="7.5"}',
'waku_filter_handle_message_duration_seconds_bucket{le="10.0"}',
'waku_filter_handle_message_duration_seconds_bucket{le="+Inf"}',
'logos_delivery_archive_messages{type="stored"}',
"logos_delivery_archive_insert_duration_seconds_sum",
"logos_delivery_archive_insert_duration_seconds_count",
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.005"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.01"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.025"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.05"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.075"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.1"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.25"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.5"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="0.75"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="1.0"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="2.5"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="5.0"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="7.5"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="10.0"}',
'logos_delivery_archive_insert_duration_seconds_bucket{le="+Inf"}',
"logos_delivery_archive_query_duration_seconds_sum",
"logos_delivery_archive_query_duration_seconds_count",
'logos_delivery_archive_query_duration_seconds_bucket{le="0.005"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.01"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.025"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.05"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.075"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.1"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.25"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.5"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="0.75"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="1.0"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="2.5"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="5.0"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="7.5"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="10.0"}',
'logos_delivery_archive_query_duration_seconds_bucket{le="+Inf"}',
"logos_delivery_filter_subscriptions",
"logos_delivery_filter_handle_message_duration_seconds_sum",
"logos_delivery_filter_handle_message_duration_seconds_count",
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.005"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.01"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.025"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.05"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.075"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.1"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.25"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.5"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="0.75"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="1.0"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="2.5"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="5.0"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="7.5"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="10.0"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="+Inf"}',
"discovery_session_lru_cache_hits_total",
"discovery_session_lru_cache_misses_total",
"discovery_session_decrypt_failures_total",
"discovery_unsolicited_messages_total",
"discovery_enr_auto_update_total",
"waku_px_peers_received_total",
"waku_px_peers_received_unknown",
"waku_px_peers_sent_total",
"waku_px_peers",
"waku_histogram_message_size_sum",
"waku_histogram_message_size_count",
'waku_histogram_message_size_bucket{le="0.0"}',
'waku_histogram_message_size_bucket{le="1.0"}',
'waku_histogram_message_size_bucket{le="3.0"}',
'waku_histogram_message_size_bucket{le="5.0"}',
'waku_histogram_message_size_bucket{le="15.0"}',
'waku_histogram_message_size_bucket{le="50.0"}',
'waku_histogram_message_size_bucket{le="75.0"}',
'waku_histogram_message_size_bucket{le="100.0"}',
'waku_histogram_message_size_bucket{le="125.0"}',
'waku_histogram_message_size_bucket{le="150.0"}',
'waku_histogram_message_size_bucket{le="500.0"}',
'waku_histogram_message_size_bucket{le="700.0"}',
'waku_histogram_message_size_bucket{le="1000.0"}',
'waku_histogram_message_size_bucket{le="+Inf"}',
"waku_lightpush_peers",
"waku_filter_peers",
"waku_store_peers",
"waku_px_peers",
"waku_dnsdisc_discovered",
"rendezvousPeerFoundTotal_total",
"reconciliation_roundtrips_sum",
"reconciliation_roundtrips_count",
"reconciliation_differences_sum",
"reconciliation_differences_count",
'reconciliation_roundtrips_bucket{le="1.0"}',
'reconciliation_roundtrips_bucket{le="2.0"}',
'reconciliation_roundtrips_bucket{le="3.0"}',
'reconciliation_roundtrips_bucket{le="5.0"}',
'reconciliation_roundtrips_bucket{le="8.0"}',
'reconciliation_roundtrips_bucket{le="13.0"}',
'reconciliation_roundtrips_bucket{le="+Inf"}',
'reconciliation_differences_bucket{le="0.0"}',
'reconciliation_differences_bucket{le="10.0"}',
'reconciliation_differences_bucket{le="50.0"}',
'reconciliation_differences_bucket{le="100.0"}',
'reconciliation_differences_bucket{le="500.0"}',
'reconciliation_differences_bucket{le="1000.0"}',
'reconciliation_differences_bucket{le="5000.0"}',
'reconciliation_differences_bucket{le="+Inf"}',
'waku_filter_handle_message_duration_seconds_bucket{le="15.0"}',
'waku_filter_handle_message_duration_seconds_bucket{le="20.0"}',
'waku_filter_handle_message_duration_seconds_bucket{le="30.0"}',
"total_messages_cached",
"waku_store_queries_total",
"logos_delivery_px_peers_received_total",
"logos_delivery_px_peers_received_unknown",
"logos_delivery_px_peers_sent_total",
"logos_delivery_px_peers",
"logos_delivery_histogram_message_size_sum",
"logos_delivery_histogram_message_size_count",
'logos_delivery_histogram_message_size_bucket{le="0.0"}',
'logos_delivery_histogram_message_size_bucket{le="1.0"}',
'logos_delivery_histogram_message_size_bucket{le="3.0"}',
'logos_delivery_histogram_message_size_bucket{le="5.0"}',
'logos_delivery_histogram_message_size_bucket{le="15.0"}',
'logos_delivery_histogram_message_size_bucket{le="50.0"}',
'logos_delivery_histogram_message_size_bucket{le="75.0"}',
'logos_delivery_histogram_message_size_bucket{le="100.0"}',
'logos_delivery_histogram_message_size_bucket{le="125.0"}',
'logos_delivery_histogram_message_size_bucket{le="150.0"}',
'logos_delivery_histogram_message_size_bucket{le="500.0"}',
'logos_delivery_histogram_message_size_bucket{le="700.0"}',
'logos_delivery_histogram_message_size_bucket{le="1000.0"}',
'logos_delivery_histogram_message_size_bucket{le="+Inf"}',
"logos_delivery_lightpush_peers",
"logos_delivery_filter_peers",
"logos_delivery_store_peers",
"logos_delivery_px_peers",
"logos_delivery_dnsdisc_discovered",
"logos_delivery_rendezvous_peer_found_total",
"logos_delivery_reconciliation_roundtrips_sum",
"logos_delivery_reconciliation_roundtrips_count",
"logos_delivery_reconciliation_differences_sum",
"logos_delivery_reconciliation_differences_count",
'logos_delivery_reconciliation_roundtrips_bucket{le="1.0"}',
'logos_delivery_reconciliation_roundtrips_bucket{le="2.0"}',
'logos_delivery_reconciliation_roundtrips_bucket{le="3.0"}',
'logos_delivery_reconciliation_roundtrips_bucket{le="5.0"}',
'logos_delivery_reconciliation_roundtrips_bucket{le="8.0"}',
'logos_delivery_reconciliation_roundtrips_bucket{le="13.0"}',
'logos_delivery_reconciliation_roundtrips_bucket{le="+Inf"}',
'logos_delivery_reconciliation_differences_bucket{le="0.0"}',
'logos_delivery_reconciliation_differences_bucket{le="10.0"}',
'logos_delivery_reconciliation_differences_bucket{le="50.0"}',
'logos_delivery_reconciliation_differences_bucket{le="100.0"}',
'logos_delivery_reconciliation_differences_bucket{le="500.0"}',
'logos_delivery_reconciliation_differences_bucket{le="1000.0"}',
'logos_delivery_reconciliation_differences_bucket{le="5000.0"}',
'logos_delivery_reconciliation_differences_bucket{le="+Inf"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="15.0"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="20.0"}',
'logos_delivery_filter_handle_message_duration_seconds_bucket{le="30.0"}',
"logos_delivery_total_messages_cached",
"logos_delivery_store_queries_total",
"mix_pool_size",
"libp2p_gossipsub_imreceiving_saved_messages_total",
"postgres_payload_size_bytes",
"logos_delivery_postgres_payload_size_bytes",
]

10
tests/waku_rln_relay/test_rln_group_manager_onchain.nim
View File

@ -102,7 +102,7 @@ suite "Onchain group manager":
# No members registered yet; metric should reflect nextFreeIndex == 0.
(waitFor manager.updateMemberCount()).isOkOr:
raiseAssert "updateMemberCount failed (initial): " & error
check waku_rln_number_registered_memberships.value() == 0.0
check logos_delivery_rln_number_registered_memberships.value() == 0.0
const credentialCount = 1
let credentials = generateCredentials(credentialCount)
@ -112,7 +112,8 @@ suite "Onchain group manager":
(waitFor manager.updateMemberCount()).isOkOr:
raiseAssert "updateMemberCount failed (after registrations): " & error
check waku_rln_number_registered_memberships.value() == float64(credentialCount)
check logos_delivery_rln_number_registered_memberships.value() ==
float64(credentialCount)
test "updateRoots: appends new on-chain root to validRoots after registration":
# basic check for the soon to be deprecated root contract function, is replaced by getRecentRoots()
@ -569,14 +570,15 @@ suite "Onchain group manager":
# invoke updateMemberCount on the success path.
manager.merkleProofCache = @[]
manager.proofPathRefreshInFlightFut = nil
waku_rln_number_registered_memberships.set(0.0) # baseline
logos_delivery_rln_number_registered_memberships.set(0.0) # baseline
let res = waitFor manager.ensureFreshMerkleProofPath()
check:
res.isOk()
manager.merkleProofCache.len > 0
waku_rln_number_registered_memberships.value() == float64(credentialCount)
logos_delivery_rln_number_registered_memberships.value() ==
float64(credentialCount)
test "ensureFreshMerkleProofPath: concurrent calls coalesce onto a single refresh future":
(waitFor manager.init()).isOkOr: