logos-delivery/docs/analysis/nocopy_summary_report.md

λ Async Copy Elimination in the Message Path — Summary Report

Deep-copy and hash-recomputation analysis of Logos Messaging under chronos / refc, and the measured result of removing them.

Logos Messaging (Nim) · logos-delivery · 2026-07-15 Nim 2.2.4 · --mm:refc · chronos 4.2.2 · Apple M4

Branch train: master → experimental/chore-nocopy-e2e-perf-harness (2a4da679) → experimental/chore-nocopy-wakumessage-refobj (3d98a28d) → experimental/chore-nocopy-wakuenvelop (a6f1065a · be6bfed6)

HTML version: nocopy_summary_report.html

---

01 — Hypothesis & Static Analysis

Hypothesis. Under --mm:refc, chronos async gates and Result/Option idioms force deep copies of every seq-carrying value they touch. A single inbound relayed WakuMessage (three heap seqs + a string) was predicted to cost on the order of 19 + F full-payload copies (F = filter peers) and 4–6 SHA-256 passes, almost all redundant. A secondary memory hypothesis was stated up front: no real gain in retained memory is expected (the copies are temporaries), but a slower heap-allocation elevation pattern on the GC side. Both were put to measurement.

Where chronos deep-copies (root mechanics)

Copy point	Mechanism	refc	ORC
Every {.async.} call, per seq/string param	params lambda-lifted into the closure-iterator env (asyncmacro.nim:445-517)	deep copy per param	elided (move)
complete(future, val)	val: T not sink; internalValue = val (asyncfutures.nim:198-202); move() degrades to copy under refc	1–2 copies	1 copy
let x = await f(...)	value() returns lent T; the bind to x copies	1 copy	1 copy
Result/Option bind-out (valueOr, ?, get)	accessors are lent/templates; the let bind of a large value copies; valueOr/? also copy an lvalue Result	1 copy per bind	usually moved

Accumulated per-message budget on the inbound relay path (static count, verified by counters)

Cost class	Sites	Count / msg
Redundant proto decodes of the same bytes	waku_relay/protocol.nim — ordered validator :543 · onRecv observer :271 · onValidated :326 · topicHandler :603 (+ onSend :341 outbound)	4 (receiver), ≈2 avoidable copies each
Async closure env captures of the decoded message	node/subscription_manager.nim — uniqueTopicHandler :72 + trace/filter/archive/sync/internal/legacyApp handlers :45–82	7 deep copies
Hash recomputation (computeMessageHash)	relay :217/:553/:571/:686 · archive :101 · filter :195/:246 · store-sync :78 · node publish :148	4–6 SHA-256 passes
Per-peer buffer copy (filter push)	waku_filter_v2/protocol.nim:170 — {.async.} by-value buffer per subscribed peer	F copies
Total	vs. theoretical minimum ≈ 3 (decode once · hash once · encode once)	≈ 19 + F copies, 4–6 hashes

Phase 1 instrumentation confirmed the static analysis before any fix: the two-node benchmark measured 6.0 decodes / 5.0 hashes per message process-wide — the receiver alone decoding the identical proto bytes 4×. At the 10/50/150 kB payload profile this amplifies a 50 kB message into ≈ 1 MB of heap traffic.

---

02 — Three Phases, One Branch Train

Each phase lives on its own local branch, chained for a PR train off master. Every phase ends with the same harness re-run, so each claim is a measured delta, not a projection.

Phase 01 — Measure first: e2e performance harness

experimental/chore-nocopy-e2e-perf-harness · 4 commits → 2a4da679

• Micro + macro benchmark (apps/benchmarks/message_path_bench.nim), deterministic fixed-seed workload.
• -d:msgPathCounters: counters inside WakuMessage.decode and computeMessageHash — zero cost when undefined.
• Baseline committed (bench_baseline.md) before touching production code.

Gate: counters must confirm ≥ 4 decodes and 4–6 hashes/msg — passed (6.0 / 5.0 aggregate).

Phase 02 — Copies become pointers: WakuMessage as ref object

experimental/chore-nocopy-wakumessage-refobj · 5 commits → 3d98a28d

• Every assignment / async capture / Result bind of a message: 3-seq deep copy → pointer + refcount. No signature changes.
• Structural ==, explicit clone(), immutable-by-convention.
• Aliasing audit of all mutation sites; 5 real fixes (publish timestamp, ensureTimestampSet, 2× RLN proof attach, postgres nil-init). ASAN clean.

Gate: decode/hash counters byte-identical to baseline (no behavior change) — passed (2/2 · 6/5 unchanged).

Phase 03 — Decode once, hash once: WakuEnvelope API break

experimental/chore-nocopy-wakuenvelop · 7 commits → a6f1065a

• WakuEnvelope = (msg · pubsubTopic · hash) as one ref; WakuRelayHandler takes the envelope; archive/filter/sync/FFI reuse envelope.hash.
• Unused onRecv observer decode deleted; onValidated/onSend decodes gated to DEBUG.
• Filter push buffer shared across peers (no per-peer copy).

Gate: receiver-side ≤ 2 decodes and exactly 1 hash per message — passed (2.0 / 1.0).

---

03 — Measurement Method

Scenario	Entry endpoint	Exit endpoint (measured)	What it isolates
micro — in-process, no network	raw proto bytes into the relay's registered ordered validator, then its topicHandler	return of the full dispatch chain (trace → archive insert → store-sync → internal event)	the receiver pipeline, low-noise (0.4–4 % variance)
macro — end-to-end	nodeA.publish() — public node API → gossipsub → loopback TCP	application-level relay handler on node B, firing only after validation, decode, dispatch and archive insert complete	the full pipeline incl. libp2p observers

Workload: fixed seed 42, payload mix 10 / 50 / 150 kB at 25 / 50 / 25 %, N = 1000 + 100 warmup, unique payload prefix + timestamp (no gossipsub dedup). Publisher flow-controlled to a 32-message window. Decode/hash counters live inside the codec and hash procs themselves — they count reality, not expectations. Peak heap via getMaxMem(); heap series sampled every 50 messages; retained delta via getOccupiedMem() after GC_fullCollect().

Final comparison ran both branch heads back-to-back in one session on the same machine (one post-refactor pass discarded for a >5 % variance load spike, rerun within bounds). Not covered by these numbers: RLN validation, filter push to remote peers, REST/FFI boundary, real network latency.

---

04 — Measured Gains — Before vs. After

"Before" = pre-Phase-2 head (2a4da679: original production code + harness). "After" = post-Phase-3 head (a6f1065a). Same-session A/B.

+86 % micro throughput (1,223 → 2,274 msg/s)	+57 % e2e throughput (225 → 353 msg/s)	−21 % peak heap (515 → 408 MB macro)

Full comparison

Metric	Before	After	Δ
micro msg/s	1,222.8	2,273.7	+85.9 %
micro p50 / p99 per msg	630 µs / 1.94 ms	338 µs / 1.09 ms	−46 % / −44 %
micro decodes / hashes per msg	2.0 / 2.0	2.0 / 1.0	hash −50 %
macro msg/s (e2e)	225.5	353.1	+56.6 %
macro p50 / p99 inter-arrival	3.55 ms / 10.7 ms	2.29 ms / 7.5 ms	−35 % / −30 %
macro decodes / hashes per msg (aggregate)	6.0 / 5.0	3.0 / 3.0	−50 % / −40 %
— receiver-side only	4 dec / 3 hash	2 dec / 1 hash	gate met
bytes decoded / hashed per 1000 msgs	390 / 325 MB	195 / 195 MB	−50 % / −40 %
peak heap getMaxMem (micro / macro)	333.6 / 514.9 MB	263.6 / 408.4 MB	−21 % both
GC collections (micro / macro)	7–8 / 10	7–8 / 10	identical
retained-mem slope, macro	≈ 12.6 MB / 100 msg	≈ 13.2 MB / 100 msg	flat (archive-driven)

Memory hypothesis — verdict

(a) No real retained-memory gain — CONFIRMED. Retention slope is identical before and after — it is the archive holding the same 1000 messages. The eliminated copies were temporaries, exactly as hypothesized.

(b) Slower heap-elevation pattern — REFUTED; level shift instead. Slope and collection counts are unchanged: under refc the transient copies were refcount-freed deterministically and never accumulated toward collection triggers. The win manifests as a −21 % peak heap and a ~65–75 MB lower level throughout the run — the copies cost CPU (memcpy + alloc/free churn) and peak footprint, not GC-cycle pressure. Quantifying raw churn would require cumulative-allocation counters or malloc profiling.

---

Conclusions & Open Items

The message path now performs one decode per trust boundary and one hash per message, carried by reference through every async gate. Verification at each phase: representative suites green (~160+ tests), ASAN clean on the dispatch path, counters byte-exact where no change was claimed.

• Per-shard payload byte gauges (waku_relay_*_msg_bytes_per_shard) now populate only at DEBUG/TRACE — decide before merge if dashboards need them.
• Archive/filter keep thin (topic, msg) compat overloads; removable later.
• Lightpush validateMessage double-encode deferred (needs PushMessageHandler signature change).
• Under a future ORC build the remaining complete()/await-bind copies persist — returning refs from async procs stays best practice.

Sources: async_copy_analysis.md · async_copy_fix_plan.md · plan_phase{1,2,3}_*.md · bench_baseline.md · phase2_mutation_audit.md · speed_gain_pre2_post3.md — all committed on the branch train.

Logos.co · λ Logos Messaging · Engineering Report · 2026-07-15