Hypothesis. Under --mm:refc, chronos async gates and Result/Option idioms force
+ deep copies of every seq-carrying value they touch. A single inbound relayed
+ WakuMessage (three heap seqs + a string) was predicted to cost on the order of
+ 19 + F full-payload copies (F = filter peers) and 4–6 SHA-256 passes,
+ almost all redundant. A secondary memory hypothesis was stated up front: no real gain in retained
+ memory is expected (the copies are temporaries), but a slower heap-allocation elevation pattern on the
+ GC side. Both were put to measurement.
| Copy point | Mechanism | refc | ORC |
|---|---|---|---|
Every {.async.} call, per seq/string param |
+ params lambda-lifted into the closure-iterator env (asyncmacro.nim:445-517) |
+ deep copy per param | elided (move) |
complete(future, val) |
+ val: T not sink; internalValue = val (asyncfutures.nim:198-202); move() degrades to copy under refc |
+ 1–2 copies | 1 copy |
let x = await f(...) |
+ value() returns lent T; the bind to x copies |
+ 1 copy | 1 copy |
Result/Option bind-out (valueOr, ?, get) |
+ accessors are lent/templates; the let bind of a large value copies; valueOr/? also copy an lvalue Result |
+ 1 copy per bind | usually moved |
| Cost class | Sites | Count / msg |
|---|---|---|
| Redundant proto decodes of the same bytes | +waku_relay/protocol.nim — ordered validator :543 · onRecv observer :271 · onValidated :326 · topicHandler :603 (+ onSend :341 outbound) |
+ 4 (receiver), ≈2 avoidable copies each |
| Async closure env captures of the decoded message | +node/subscription_manager.nim — uniqueTopicHandler :72 + trace/filter/archive/sync/internal/legacyApp handlers :45–82 |
+ 7 deep copies |
Hash recomputation (computeMessageHash) |
+ relay :217/:553/:571/:686 · archive :101 · filter :195/:246 · store-sync :78 · node publish :148 | +4–6 SHA-256 passes |
| Per-peer buffer copy (filter push) | +waku_filter_v2/protocol.nim:170 — {.async.} by-value buffer per subscribed peer |
+ F copies |
| Total | vs. theoretical minimum ≈ 3 (decode once · hash once · encode once) | +≈ 19 + F copies, 4–6 hashes |
Phase 1 instrumentation confirmed the static analysis before any fix: the two-node + benchmark measured 6.0 decodes / 5.0 hashes per message process-wide — the receiver alone decoding the + identical proto bytes 4×. At the 10/50/150 kB payload profile this amplifies a 50 kB message + into ≈ 1 MB of heap traffic.
+Each phase lives on its own local branch, chained for a PR train off master.
+ Every phase ends with the same harness re-run, so each claim is a measured delta, not a projection.
apps/benchmarks/message_path_bench.nim), deterministic fixed-seed workload.-d:msgPathCounters: counters inside WakuMessage.decode and computeMessageHash — zero cost when undefined.bench_baseline.md) before touching production code.ref object==, explicit clone(), immutable-by-convention.ensureTimestampSet, 2× RLN proof attach, postgres nil-init). ASAN clean.WakuEnvelope = (msg · pubsubTopic · hash) as one ref; WakuRelayHandler takes the envelope; archive/filter/sync/FFI reuse envelope.hash.| Scenario | Entry endpoint | Exit endpoint (measured) | What it isolates |
|---|---|---|---|
| micro — in-process, no network | +raw proto bytes into the relay’s registered ordered validator, then its topicHandler | +return of the full dispatch chain (trace → archive insert → store-sync → internal event) | +the receiver pipeline, low-noise (0.4–4 % variance) | +
| macro — end-to-end | +nodeA.publish() — public node API → gossipsub → loopback TCP |
+ application-level relay handler on node B, firing only after validation, decode, dispatch and archive insert complete | +the full pipeline incl. libp2p observers | +
Workload: fixed seed 42, payload mix 10 / 50 / 150 kB at 25 / 50 / 25 %, N = 1000 + 100 warmup,
+ unique payload prefix + timestamp (no gossipsub dedup). Publisher flow-controlled to a 32-message window.
+ Decode/hash counters live inside the codec and hash procs themselves — they count reality, not expectations.
+ Peak heap via getMaxMem(); heap series sampled every 50 messages; retained delta via
+ getOccupiedMem() after GC_fullCollect().
Final comparison ran both branch heads back-to-back in one session on the same machine + (one post-refactor pass discarded for a >5 % variance load spike, rerun within bounds). Not covered by + these numbers: RLN validation, filter push to remote peers, REST/FFI boundary, real network latency.
+“Before” = pre-Phase-2 head (2a4da679: original production code + harness).
+ “After” = post-Phase-3 head (a6f1065a). Same-session A/B.
| Metric | Before | After | Δ |
|---|---|---|---|
| micro msg/s | 1,222.8 | 2,273.7 | +85.9 % |
| micro p50 / p99 per msg | 630 µs / 1.94 ms | 338 µs / 1.09 ms | −46 % / −44 % |
| micro decodes / hashes per msg | 2.0 / 2.0 | 2.0 / 1.0 | hash −50 % |
| macro msg/s (e2e) | 225.5 | 353.1 | +56.6 % |
| macro p50 / p99 inter-arrival | 3.55 ms / 10.7 ms | 2.29 ms / 7.5 ms | −35 % / −30 % |
| macro decodes / hashes per msg (aggregate) | 6.0 / 5.0 | 3.0 / 3.0 | −50 % / −40 % |
| — receiver-side only | 4 dec / 3 hash | 2 dec / 1 hash | gate met |
| bytes decoded / hashed per 1000 msgs | 390 / 325 MB | 195 / 195 MB | −50 % / −40 % |
peak heap getMaxMem (micro / macro) | 333.6 / 514.9 MB | 263.6 / 408.4 MB | −21 % both |
| GC collections (micro / macro) | 7–8 / 10 | 7–8 / 10 | identical |
| retained-mem slope, macro | ≈ 12.6 MB / 100 msg | ≈ 13.2 MB / 100 msg | flat (archive-driven) |
Retention slope is identical before and after — it is the archive holding the same 1000 messages. + The eliminated copies were temporaries, exactly as hypothesized.
+Slope and collection counts are unchanged: under refc the transient copies were refcount-freed + deterministically and never accumulated toward collection triggers. The win manifests as a + −21 % peak heap and a ~65–75 MB lower level throughout the run — the copies cost CPU + (memcpy + alloc/free churn) and peak footprint, not GC-cycle pressure. Quantifying raw churn would + require cumulative-allocation counters or malloc profiling.
+The message path now performs one decode per trust boundary and one hash per message, + carried by reference through every async gate. Verification at each phase: representative suites green + (~160+ tests), ASAN clean on the dispatch path, counters byte-exact where no change was claimed.
+waku_relay_*_msg_bytes_per_shard) now populate only at DEBUG/TRACE — decide before merge if dashboards need them.(topic, msg) compat overloads; removable later.validateMessage double-encode deferred (needs PushMessageHandler signature change).complete()/await-bind copies persist — returning refs from async procs stays best practice.Sources: docs/analysis/async_copy_analysis.md · async_copy_fix_plan.md · plan_phase{1,2,3}_*.md · + bench_baseline.md · phase2_mutation_audit.md · speed_gain_pre2_post3.md — all committed on the branch train. Local branches only; nothing pushed.
+