diff --git a/docs/analysis/nocopy_summary_report.html b/docs/analysis/nocopy_summary_report.html new file mode 100644 index 000000000..82d7f2a32 --- /dev/null +++ b/docs/analysis/nocopy_summary_report.html @@ -0,0 +1,358 @@ + + + + + +Logos Messaging — Async Copy Elimination Report + + + + + +
+
+
λ
+

Async Copy Elimination in the Message Path

+
Deep-copy and hash-recomputation analysis of Logos Messaging under chronos / refc, + and the measured result of removing them — engineering summary report.
+
+ LOGOS MESSAGING (NIM)  ·  logos-delivery  ·  2026-07-15 +  ·  Nim 2.2.4 · --mm:refc · chronos 4.2.2 · Apple M4 +
+
+ master → experimental/chore-nocopy-e2e-perf-harness (2a4da679) + → experimental/chore-nocopy-wakumessage-refobj (3d98a28d) + → experimental/chore-nocopy-wakuenvelop (a6f1065a · be6bfed6) +
+
+
+ + +
+
+
01
+

Hypothesis & Static Analysis

+ +

Hypothesis. Under --mm:refc, chronos async gates and Result/Option idioms force + deep copies of every seq-carrying value they touch. A single inbound relayed + WakuMessage (three heap seqs + a string) was predicted to cost on the order of + 19 + F full-payload copies (F = filter peers) and 4–6 SHA-256 passes, + almost all redundant. A secondary memory hypothesis was stated up front: no real gain in retained + memory is expected (the copies are temporaries), but a slower heap-allocation elevation pattern on the + GC side. Both were put to measurement.

+ +

Where chronos deep-copies (root mechanics)

+
+ + + + + + + + + + + + + +
Copy pointMechanismrefcORC
Every {.async.} call, per seq/string paramparams lambda-lifted into the closure-iterator env (asyncmacro.nim:445-517)deep copy per paramelided (move)
complete(future, val)val: T not sink; internalValue = val (asyncfutures.nim:198-202); move() degrades to copy under refc1–2 copies1 copy
let x = await f(...)value() returns lent T; the bind to x copies1 copy1 copy
Result/Option bind-out (valueOr, ?, get)accessors are lent/templates; the let bind of a large value copies; valueOr/? also copy an lvalue Result1 copy per bindusually moved
+ +

Accumulated per-message budget on the inbound relay path (static count, verified by counters)

+
+ + + + + + + + + + + + + + + +
Cost classSitesCount / msg
Redundant proto decodes of the same byteswaku_relay/protocol.nim — ordered validator :543 · onRecv observer :271 · onValidated :326 · topicHandler :603 (+ onSend :341 outbound)4 (receiver), ≈2 avoidable copies each
Async closure env captures of the decoded messagenode/subscription_manager.nim — uniqueTopicHandler :72 + trace/filter/archive/sync/internal/legacyApp handlers :45–827 deep copies
Hash recomputation (computeMessageHash)relay :217/:553/:571/:686 · archive :101 · filter :195/:246 · store-sync :78 · node publish :1484–6 SHA-256 passes
Per-peer buffer copy (filter push)waku_filter_v2/protocol.nim:170 — {.async.} by-value buffer per subscribed peerF copies
Totalvs. theoretical minimum ≈ 3 (decode once · hash once · encode once)≈ 19 + F copies, 4–6 hashes
+

Phase 1 instrumentation confirmed the static analysis before any fix: the two-node + benchmark measured 6.0 decodes / 5.0 hashes per message process-wide — the receiver alone decoding the + identical proto bytes 4×. At the 10/50/150 kB payload profile this amplifies a 50 kB message + into ≈ 1 MB of heap traffic.

+
+
+ + +
+
+
02
+

Three Phases, One Branch Train

+

Each phase lives on its own local branch, chained for a PR train off master. + Every phase ends with the same harness re-run, so each claim is a measured delta, not a projection.

+
+
+
PHASE 01 — Measure first
+

E2E performance harness

+
experimental/chore-nocopy-e2e-perf-harness · 4 commits → 2a4da679
+
    +
  • Micro + macro benchmark (apps/benchmarks/message_path_bench.nim), deterministic fixed-seed workload.
    • +
  • -d:msgPathCounters: counters inside WakuMessage.decode and computeMessageHash — zero cost when undefined.
    • +
  • Baseline committed (bench_baseline.md) before touching production code.
    • +
+
Gate: counters must confirm ≥ 4 decodes and 4–6 hashes/msg — passed (6.0 / 5.0 aggregate).
+
+
+
PHASE 02 — Copies become pointers
+

WakuMessage as ref object

+
experimental/chore-nocopy-wakumessage-refobj · 5 commits → 3d98a28d
+
    +
  • Every assignment / async capture / Result bind of a message: 3-seq deep copy → pointer + refcount. No signature changes.
    • +
  • Structural ==, explicit clone(), immutable-by-convention.
    • +
  • Aliasing audit of all mutation sites; 5 real fixes (publish timestamp, ensureTimestampSet, 2× RLN proof attach, postgres nil-init). ASAN clean.
    • +
+
Gate: decode/hash counters byte-identical to baseline (no behavior change) — passed (2/2 · 6/5 unchanged).
+
+
+
PHASE 03 — Decode once, hash once
+

WakuEnvelope API break

+
experimental/chore-nocopy-wakuenvelop · 7 commits → a6f1065a
+
    +
  • WakuEnvelope = (msg · pubsubTopic · hash) as one ref; WakuRelayHandler takes the envelope; archive/filter/sync/FFI reuse envelope.hash.
    • +
  • Unused onRecv observer decode deleted; onValidated/onSend decodes gated to DEBUG.
    • +
  • Filter push buffer shared across peers (no per-peer copy).
    • +
+
Gate: receiver-side ≤ 2 decodes and exactly 1 hash per message — passed (2.0 / 1.0).
+
+
+
+
+ + +
+
+
03
+

Measurement Method

+
+ + + + + + + + + + + + + +
ScenarioEntry endpointExit endpoint (measured)What it isolates
micro — in-process, no networkraw proto bytes into the relay’s registered ordered validator, then its topicHandlerreturn of the full dispatch chain (trace → archive insert → store-sync → internal event)the receiver pipeline, low-noise (0.4–4 % variance)
macro — end-to-endnodeA.publish() — public node API → gossipsub → loopback TCPapplication-level relay handler on node B, firing only after validation, decode, dispatch and archive insert completethe full pipeline incl. libp2p observers
+

Workload: fixed seed 42, payload mix 10 / 50 / 150 kB at 25 / 50 / 25 %, N = 1000 + 100 warmup, + unique payload prefix + timestamp (no gossipsub dedup). Publisher flow-controlled to a 32-message window. + Decode/hash counters live inside the codec and hash procs themselves — they count reality, not expectations. + Peak heap via getMaxMem(); heap series sampled every 50 messages; retained delta via + getOccupiedMem() after GC_fullCollect().

+

Final comparison ran both branch heads back-to-back in one session on the same machine + (one post-refactor pass discarded for a >5 % variance load spike, rerun within bounds). Not covered by + these numbers: RLN validation, filter push to remote peers, REST/FFI boundary, real network latency.

+
+
+ + +
+
+
04
+

Measured Gains — Before vs. After

+

“Before” = pre-Phase-2 head (2a4da679: original production code + harness). + “After” = post-Phase-3 head (a6f1065a). Same-session A/B.

+ +
+
+86 %
micro throughput
1,223 → 2,274 msg/s
+
+57 %
e2e (macro) throughput
225 → 353 msg/s
+
−21 %
peak heap (both scenarios)
515 → 408 MB macro
+
4 → 2
receiver decodes / msg
hashes: receiver 3 → 1
+
+ +
+ Before (pre-Phase-2) + After (post-Phase-3) + bars are supplementary — full values in the tables below +
+ +
+
Throughput (msg/s, higher is better)
+
+
micro — before
1,223
+
micro — after
2,274
+
+
+
macro e2e — before
225
+
macro e2e — after
353
+
+
+ +
+
Redundant work per message (macro, process aggregate — lower is better)
+
+
proto decodes — before
6.0
+
proto decodes — after
3.0
+
+
+
SHA-256 passes — before
5.0
+
SHA-256 passes — after
3.0
+
+
+
bytes hashed — before
325 MB / 1000 msgs
+
bytes hashed — after
195 MB / 1000 msgs
+
+
+ +

Full comparison

+
+ + + + + + + + + + + + +
MetricBeforeAfterΔ
micro msg/s1,222.82,273.7+85.9 %
micro p50 / p99 per msg630 µs / 1.94 ms338 µs / 1.09 ms−46 % / −44 %
micro decodes / hashes per msg2.0 / 2.02.0 / 1.0hash −50 %
macro msg/s (e2e)225.5353.1+56.6 %
macro p50 / p99 inter-arrival3.55 ms / 10.7 ms2.29 ms / 7.5 ms−35 % / −30 %
macro decodes / hashes per msg (aggregate)6.0 / 5.03.0 / 3.0−50 % / −40 %
  — receiver-side only4 dec / 3 hash2 dec / 1 hashgate met
bytes decoded / hashed per 1000 msgs390 / 325 MB195 / 195 MB−50 % / −40 %
peak heap getMaxMem (micro / macro)333.6 / 514.9 MB263.6 / 408.4 MB−21 % both
GC collections (micro / macro)7–8 / 107–8 / 10identical
retained-mem slope, macro≈ 12.6 MB / 100 msg≈ 13.2 MB / 100 msgflat (archive-driven)
+ +

Memory hypothesis — verdict

+
+
+
CONFIRMED
+

(a) No real retained-memory gain

+

Retention slope is identical before and after — it is the archive holding the same 1000 messages. + The eliminated copies were temporaries, exactly as hypothesized.

+
+
+
REFUTED — LEVEL SHIFT INSTEAD
+

(b) Slower heap-elevation pattern

+

Slope and collection counts are unchanged: under refc the transient copies were refcount-freed + deterministically and never accumulated toward collection triggers. The win manifests as a + −21 % peak heap and a ~65–75 MB lower level throughout the run — the copies cost CPU + (memcpy + alloc/free churn) and peak footprint, not GC-cycle pressure. Quantifying raw churn would + require cumulative-allocation counters or malloc profiling.

+
+
+
+
+ + +
+
+

Conclusions & Open Items

+

The message path now performs one decode per trust boundary and one hash per message, + carried by reference through every async gate. Verification at each phase: representative suites green + (~160+ tests), ASAN clean on the dispatch path, counters byte-exact where no change was claimed.

+ +

Sources: docs/analysis/async_copy_analysis.md · async_copy_fix_plan.md · plan_phase{1,2,3}_*.md · + bench_baseline.md · phase2_mutation_audit.md · speed_gain_pre2_post3.md — all committed on the branch train. Local branches only; nothing pushed.

+
Logos.coλ Logos Messaging · Engineering Report · 2026-07-15
+
+
+ + +