Hypothesis. Under --mm:refc, chronos async gates and Result/Option idioms force
deep copies of every seq-carrying value they touch. A single inbound relayed
WakuMessage (three heap seqs + a string) was predicted to cost on the order of
19 + F full-payload copies (F = filter peers) and 4–6 SHA-256 passes,
almost all redundant. A secondary memory hypothesis was stated up front: no real gain in retained
memory is expected (the copies are temporaries), but a slower heap-allocation elevation pattern on the
GC side. Both were put to measurement.
| Copy point | Mechanism | refc | ORC |
|---|---|---|---|
Every {.async.} call, per seq/string param |
params lambda-lifted into the closure-iterator env (asyncmacro.nim:445-517) |
deep copy per param | elided (move) |
complete(future, val) |
val: T not sink; internalValue = val (asyncfutures.nim:198-202); move() degrades to copy under refc |
1–2 copies | 1 copy |
let x = await f(...) |
value() returns lent T; the bind to x copies |
1 copy | 1 copy |
Result/Option bind-out (valueOr, ?, get) |
accessors are lent/templates; the let bind of a large value copies; valueOr/? also copy an lvalue Result |
1 copy per bind | usually moved |
| Cost class | Sites | Count / msg |
|---|---|---|
| Redundant proto decodes of the same bytes | waku_relay/protocol.nim — ordered validator :543 · onRecv observer :271 · onValidated :326 · topicHandler :603 (+ onSend :341 outbound) |
4 (receiver), ≈2 avoidable copies each |
| Async closure env captures of the decoded message | node/subscription_manager.nim — uniqueTopicHandler :72 + trace/filter/archive/sync/internal/legacyApp handlers :45–82 |
7 deep copies |
Hash recomputation (computeMessageHash) |
relay :217/:553/:571/:686 · archive :101 · filter :195/:246 · store-sync :78 · node publish :148 | 4–6 SHA-256 passes |
| Per-peer buffer copy (filter push) | waku_filter_v2/protocol.nim:170 — {.async.} by-value buffer per subscribed peer |
F copies |
| Total | vs. theoretical minimum ≈ 3 (decode once · hash once · encode once) | ≈ 19 + F copies, 4–6 hashes |
Phase 1 instrumentation confirmed the static analysis before any fix: the two-node benchmark measured 6.0 decodes / 5.0 hashes per message process-wide — the receiver alone decoding the identical proto bytes 4×. At the 10/50/150 kB payload profile this amplifies a 50 kB message into ≈ 1 MB of heap traffic.
Each phase lives on its own local branch, chained for a PR train off master.
Every phase ends with the same harness re-run, so each claim is a measured delta, not a projection.
apps/benchmarks/message_path_bench.nim), deterministic fixed-seed workload.-d:msgPathCounters: counters inside WakuMessage.decode and computeMessageHash — zero cost when undefined.bench_baseline.md) before touching production code.ref object==, explicit clone(), immutable-by-convention.ensureTimestampSet, 2× RLN proof attach, postgres nil-init). ASAN clean.WakuEnvelope = (msg · pubsubTopic · hash) as one ref; WakuRelayHandler takes the envelope; archive/filter/sync/FFI reuse envelope.hash.| Scenario | Entry endpoint | Exit endpoint (measured) | What it isolates |
|---|---|---|---|
| micro — in-process, no network | raw proto bytes into the relay’s registered ordered validator, then its topicHandler | return of the full dispatch chain (trace → archive insert → store-sync → internal event) | the receiver pipeline, low-noise (0.4–4 % variance) |
| macro — end-to-end | nodeA.publish() — public node API → gossipsub → loopback TCP |
application-level relay handler on node B, firing only after validation, decode, dispatch and archive insert complete | the full pipeline incl. libp2p observers |
Workload: fixed seed 42, payload mix 10 / 50 / 150 kB at 25 / 50 / 25 %, N = 1000 + 100 warmup,
unique payload prefix + timestamp (no gossipsub dedup). Publisher flow-controlled to a 32-message window.
Decode/hash counters live inside the codec and hash procs themselves — they count reality, not expectations.
Peak heap via getMaxMem(); heap series sampled every 50 messages; retained delta via
getOccupiedMem() after GC_fullCollect().
Final comparison ran both branch heads back-to-back in one session on the same machine (one post-refactor pass discarded for a >5 % variance load spike, rerun within bounds). Not covered by these numbers: RLN validation, filter push to remote peers, REST/FFI boundary, real network latency.
“Before” = pre-Phase-2 head (2a4da679: original production code + harness).
“After” = post-Phase-3 head (a6f1065a). Same-session A/B.
| Metric | Before | After | Δ |
|---|---|---|---|
| micro msg/s | 1,222.8 | 2,273.7 | +85.9 % |
| micro p50 / p99 per msg | 630 µs / 1.94 ms | 338 µs / 1.09 ms | −46 % / −44 % |
| micro decodes / hashes per msg | 2.0 / 2.0 | 2.0 / 1.0 | hash −50 % |
| macro msg/s (e2e) | 225.5 | 353.1 | +56.6 % |
| macro p50 / p99 inter-arrival | 3.55 ms / 10.7 ms | 2.29 ms / 7.5 ms | −35 % / −30 % |
| macro decodes / hashes per msg (aggregate) | 6.0 / 5.0 | 3.0 / 3.0 | −50 % / −40 % |
| — receiver-side only | 4 dec / 3 hash | 2 dec / 1 hash | gate met |
| bytes decoded / hashed per 1000 msgs | 390 / 325 MB | 195 / 195 MB | −50 % / −40 % |
peak heap getMaxMem (micro / macro) | 333.6 / 514.9 MB | 263.6 / 408.4 MB | −21 % both |
| GC collections (micro / macro) | 7–8 / 10 | 7–8 / 10 | identical |
| retained-mem slope, macro | ≈ 12.6 MB / 100 msg | ≈ 13.2 MB / 100 msg | flat (archive-driven) |
Retention slope is identical before and after — it is the archive holding the same 1000 messages. The eliminated copies were temporaries, exactly as hypothesized.
Slope and collection counts are unchanged: under refc the transient copies were refcount-freed deterministically and never accumulated toward collection triggers. The win manifests as a −21 % peak heap and a ~65–75 MB lower level throughout the run — the copies cost CPU (memcpy + alloc/free churn) and peak footprint, not GC-cycle pressure. Quantifying raw churn would require cumulative-allocation counters or malloc profiling.
The message path now performs one decode per trust boundary and one hash per message, carried by reference through every async gate. Verification at each phase: representative suites green (~160+ tests), ASAN clean on the dispatch path, counters byte-exact where no change was claimed.
waku_relay_*_msg_bytes_per_shard) now populate only at DEBUG/TRACE — decide before merge if dashboards need them.(topic, msg) compat overloads; removable later.validateMessage double-encode deferred (needs PushMessageHandler signature change).complete()/await-bind copies persist — returning refs from async procs stays best practice.Sources: docs/analysis/async_copy_analysis.md · async_copy_fix_plan.md · plan_phase{1,2,3}_*.md · bench_baseline.md · phase2_mutation_audit.md · speed_gain_pre2_post3.md — all committed on the branch train. Local branches only; nothing pushed.