Igor Sirotin 8125cd0262
refactor(metrics): give every metric a logos_delivery_ prefix (#4074)
* refactor(metrics): prefix node metrics with logos_delivery_

Every metric the node exports now starts with logos_delivery_. There was no
prefix mechanism before: nim-metrics derives the exported name from the Nim
identifier, no declaration passed an explicit `name = "..."`, and the waku_
convention was maintained by hand -- 69 of the 80 node metrics followed it and
11 did not (query_count, query_time_secs, event_loop_load,
event_loop_accumulated_lag_secs, postgres_payload_size_bytes, reconciliation_*,
total_* and the camelCase rendezvousPeerFoundTotal).

Identifiers are renamed rather than given a `name = "..."` argument, keeping the
invariant that the Nim identifier is the exported name and letting the compiler
check every call site.

rendezvousPeerFoundTotal becomes logos_delivery_rendezvous_peer_found: it was the
only camelCase metric, and the trailing Total was redundant since nim-metrics
already appends _total to counters at exposition time.

library/ is untouched on purpose -- `proc waku_version()` in
kernel_api/debug_node_api.nim is the exported libwaku C ABI symbol, not the gauge
of the same name in node_telemetry.nim.

BREAKING CHANGE: metric names change. Dashboards, alert rules and recording rules
that reference waku_* must be updated; see docs/operators/how-to/monitor.md.

* refactor(metrics): prefix auxiliary app metrics with logos_delivery_

Applies the same prefix to the tools shipped from this repo: liteprotocoltester
(lpt_*), networkmonitor (networkmonitor_*), chat2bridge (chat2_*) and the
lightpush_mix example (lp_mix_*).

These tools are not the delivery node and already had their own consistent
prefixes, so this commit is separable from the node rename if the intent was to
namespace only the node itself.

* chore(metrics): query old and new metric names in Grafana dashboards

212 expressions across 9 dashboards now match both the waku_* and the
logos_delivery_* spelling, so panels keep working across the upgrade and over
historical data:

  sum by (type)((increase(waku_node_errors_total{...}[$__rate_interval])
              or increase(logos_delivery_node_errors_total{...}[$__rate_interval])))

The `or` is placed around the leaf, inside every aggregation. That depth is
load-bearing: `or` keeps its right operand only for label sets absent from the
left, so `sum by (type)(old) or sum by (type)(new)` aggregates each half of the
fleet separately and then discards the right one entirely -- silently dropping
every already-upgraded node. 36 panels here collapse `instance`.

Measured against a local Prometheus scraping two targets, one exporting old names
at 10/s and one exporting new names at 20/s (truth 30/s): union outside the
aggregation gives 10, union around the leaf gives 30.

Where the leaf sits in a range vector the whole call is duplicated, since
`(a or b)[5m]` is not valid PromQL.

Once every scraped node runs a release with the new names and the old samples
have aged out of retention, the `or` half can be deleted.

* test(e2e): expect logos_delivery_-prefixed metric names

The e2e suite asserts against a live /metrics endpoint, which serves only the new
names, so these are replaced rather than unioned. libp2p_* entries are unchanged.

* docs(operators): document the logos_delivery_ metric prefix

Records that every metric the node exports is prefixed, that dependency metrics
(libp2p_*, nim_gc_*, process_*) keep their own names, and shows where the `or`
has to sit if operators maintain their own dashboards or alert rules.

* refactor(metrics): name the store fleet metrics after store, not relay

logos_delivery_relay_fleet_store_msg_size_bytes and _msg_count are declared in
waku_store/protocol_metrics.nim and recorded by the store client, but carried a
relay prefix. Renamed to logos_delivery_store_fleet_msg_size_bytes and
logos_delivery_store_fleet_msg_count.

The dashboard keeps matching the old exported name, which was
waku_relay_fleet_store_*.

Note that both metrics are wrong independently of their name, see the PR
description.
2026-07-29 18:56:50 +01:00

264 lines
9.2 KiB
Nim

{.push raises: [].}
import
std/net,
chronos,
chronicles,
metrics,
results,
stew/byteutils,
eth/keys,
eth/p2p/discoveryv5/enr,
libp2p/crypto/crypto,
libp2p/protocols/ping,
libp2p/protocols/pubsub/gossipsub,
libp2p/protocols/pubsub/pubsub,
libp2p/protocols/pubsub/rpc/messages,
libp2p/builders,
libp2p/transports/tcptransport,
libp2p/transports/wstransport,
libp2p/utility,
brokers/broker_context
import
logos_delivery/waku/[
waku_relay,
waku_core,
waku_core/topics/sharding,
waku_filter_v2,
waku_archive,
waku_store_sync,
rln,
node/waku_node,
node/subscription_manager,
node/peer_manager,
]
import logos_delivery/api/events/kernel_events # MessageSeenEvent
export waku_relay.WakuRelayHandler
logScope:
topics = "waku node relay api"
## Waku relay
proc getTopicOfSubscriptionEvent(
node: WakuNode, subscription: SubscriptionEvent
): Result[(PubsubTopic, Opt[ContentTopic]), string] =
case subscription.kind
of ContentSub, ContentUnsub:
if node.wakuAutoSharding.isSome():
let shard = node.wakuAutoSharding.get().getShard((subscription.topic)).valueOr:
return err("Autosharding error: " & error)
return ok(($shard, Opt.some(subscription.topic)))
else:
return
err("Static sharding is used, relay subscriptions must specify a pubsub topic")
of PubsubSub, PubsubUnsub:
return ok((subscription.topic, Opt.none(ContentTopic)))
else:
return err("Unsupported subscription type in relay getTopicOfSubscriptionEvent")
proc subscribe*(
node: WakuNode, subscription: SubscriptionEvent, handler: WakuRelayHandler
): Result[void, string] =
## Subscribes to a PubSub or Content topic. Triggers handler when receiving messages on
## this topic. WakuRelayHandler is a method that takes a topic and a Waku message.
## If `handler` is nil, the API call will subscribe to the topic in the relay mesh
## but no app handler will be registered at this time (it can be registered later with
## another call to this proc for the same gossipsub topic).
if isNil(node.wakuRelay):
error "Invalid API call to `subscribe`. WakuRelay not mounted."
return err("Invalid API call to `subscribe`. WakuRelay not mounted.")
let (pubsubTopic, _) = getTopicOfSubscriptionEvent(node, subscription).valueOr:
error "Failed to decode subscription event", error = error
return err("Failed to decode subscription event: " & error)
# strict version
#if contentTopicOp.isSome():
# return
# node.subscriptionManager.subscribe(pubsubTopic, contentTopicOp.get(), handler)
return node.subscriptionManager.subscribeShard(pubsubTopic, handler)
proc unsubscribe*(
node: WakuNode, subscription: SubscriptionEvent
): Result[void, string] =
## Unsubscribes from a specific PubSub or Content topic.
## This will both unsubscribe from the relay mesh and remove the app handler, if any.
## NOTE: This works because using MAPI and Kernel API at the same time is unsupported.
if isNil(node.wakuRelay):
error "Invalid API call to `unsubscribe`. WakuRelay not mounted."
return err("Invalid API call to `unsubscribe`. WakuRelay not mounted.")
let (pubsubTopic, _) = getTopicOfSubscriptionEvent(node, subscription).valueOr:
error "Failed to decode unsubscribe event", error = error
return err("Failed to decode unsubscribe event: " & error)
# strict version
#if contentTopicOp.isSome():
# return node.subscriptionManager.unsubscribe(pubsubTopic, contentTopicOp.get())
return node.subscriptionManager.unsubscribeAll(pubsubTopic)
proc isSubscribed*(
node: WakuNode, subscription: SubscriptionEvent
): Result[bool, string] =
if node.wakuRelay.isNil():
error "Invalid API call to `isSubscribed`. WakuRelay not mounted."
return err("Invalid API call to `isSubscribed`. WakuRelay not mounted.")
let (pubsubTopic, contentTopicOp) = getTopicOfSubscriptionEvent(node, subscription).valueOr:
error "Failed to decode subscription event", error = error
return err("Failed to decode subscription event: " & error)
return ok(node.wakuRelay.isSubscribed(pubsubTopic))
proc publish*(
node: WakuNode, pubsubTopicOp: Opt[PubsubTopic], message: WakuMessage
): Future[Result[int, string]] {.async, gcsafe.} =
## Publish a `WakuMessage`. Pubsub topic contains; none, a named or static shard.
## `WakuMessage` should contain a `contentTopic` field for light node functionality.
## It is also used to determine the shard.
if node.wakuRelay.isNil():
let msg =
"Invalid API call to `publish`. WakuRelay not mounted. Try `lightpush` instead."
error "publish error", err = msg
# TODO: Improve error handling
return err(msg)
let pubsubTopic = pubsubTopicOp.valueOr:
if node.wakuAutoSharding.isNone():
return err("Pubsub topic must be specified when static sharding is enabled.")
node.wakuAutoSharding.get().getShard(message.contentTopic).valueOr:
let msg = "Autosharding error: " & error
return err(msg)
let numPeers = (await node.wakuRelay.publish(pubsubTopic, message)).valueOr:
warn "waku.relay did not publish", error = error
# Todo: If NoPeersToPublish, we might want to return ok(0) instead!!!
return err("publish failed in relay: " & $error)
notice "waku.relay published",
peerId = node.peerId,
pubsubTopic = pubsubTopic,
msg_hash = pubsubTopic.computeMessageHash(message).to0xHex(),
publishTime = getNowInNanosecondTime(),
numPeers = numPeers
# TODO: investigate if we can return error in case numPeers is 0
ok(numPeers)
proc mountRelay*(
node: WakuNode,
peerExchangeHandler = Opt.none(RoutingRecordsHandler),
maxMessageSize = int(DefaultMaxWakuMessageSize),
): Future[Result[void, string]] {.async.} =
if not node.wakuRelay.isNil():
error "wakuRelay already mounted, skipping"
return err("wakuRelay already mounted, skipping")
## The default relay topics is the union of all configured topics plus default PubsubTopic(s)
info "mounting relay protocol"
node.wakuRelay = WakuRelay.new(node.switch, maxMessageSize).valueOr:
error "failed mounting relay protocol", error = error
return err("failed mounting relay protocol: " & error)
## Add peer exchange handler
if peerExchangeHandler.isSome():
node.wakuRelay.parameters.enablePX = true
# Feature flag for peer exchange in nim-libp2p
node.wakuRelay.routingRecordsHandler.add(peerExchangeHandler.get())
if node.started:
await node.wakuRelay.start()
await node.reconnectRelayPeers()
node.switch.mount(node.wakuRelay, protocolMatcher(WakuRelayCodec))
info "relay mounted successfully"
return ok()
## Waku RLN Relay
proc setRlnValidator*(
node: WakuNode,
rlnConf: WakuRlnConfig,
spamHandler = Opt.none(SpamHandler),
registrationHandler = Opt.none(RegistrationHandler),
) {.async.} =
info "setting rln validator"
let rln = (await Rln.new(rlnConf, registrationHandler)).valueOr:
raise newException(CatchableError, "failed to set rln validator: " & error)
if (rlnConf.userMessageLimit > rln.groupManager.rlnRelayMaxMessageLimit):
error "rln-user-message-limit can't exceed the MAX_MESSAGE_LIMIT in the rln contract"
node.rln = rln
if node.wakuRelay.isNil():
info "WakuRelay not mounted; RLN validator not set"
return
## Bridges RLN's protocol-agnostic message validation into a relay
## (gossipsub) validator. The core decision is made by
## `validateMessageAndUpdateLog`; this maps the result to
## `pubsub.ValidationResult` so the validator can be installed on
## WakuRelay's validator chain.
proc validator(
topic: string, message: WakuMessage
): Future[pubsub.ValidationResult] {.async.} =
trace "rln-relay topic validator is called"
rln.clearNullifierLog()
let msgProof = RateLimitProof.init(message.proof).valueOr:
trace "rln validator reject", error = error
return pubsub.ValidationResult.Reject
# validate the message and update log
let validationRes = await rln.validateMessageAndUpdateLog(message)
let
proof = byteutils.toHex(msgProof.proof)
root = inHex(msgProof.merkleRoot)
shareX = inHex(msgProof.shareX)
shareY = inHex(msgProof.shareY)
nullifier = inHex(msgProof.nullifier)
case validationRes
of Valid:
trace "message validity is verified, relaying",
proof = proof,
root = root,
shareX = shareX,
shareY = shareY,
nullifier = nullifier
logos_delivery_rln_valid_messages_total.inc(labelValues = [topic])
return pubsub.ValidationResult.Accept
of Invalid:
trace "message validity could not be verified, discarding",
proof = proof,
root = root,
shareX = shareX,
shareY = shareY,
nullifier = nullifier
return pubsub.ValidationResult.Reject
of Spam:
trace "A spam message is found! yay! discarding:",
proof = proof,
root = root,
shareX = shareX,
shareY = shareY,
nullifier = nullifier
if spamHandler.isSome():
let handler = spamHandler.get()
handler(message)
return pubsub.ValidationResult.Reject
# register rln validator as default validator
info "Registering RLN validator"
node.wakuRelay.addValidator(validator, RlnValidatorErrorMsg)