nim-sds/tests/test_snapshot_codec.nim

236 lines
7.6 KiB
Nim
Raw Normal View History

refactor(persistence): snapshot-based interface (5 procs, atomic per-op) (#72) * feat: propagate persistence backend errors via Result The Persistence contract previously returned `Future[void]` for writes and `Future[ChannelSnapshot]` for the loader, with `raises: []`. Backends had no way to report a failure, so a failed write or a failed/partial read was silently swallowed — and on the read path a mid-scan failure could bootstrap a *truncated* channel snapshot, corrupting the rebuilt bloom filter and lamport clock across a restart. Make every contract field Result-returning: * mutating ops -> Future[Result[void, string]] * loadAllForChannel -> Future[Result[ChannelSnapshot, string]] The backend-supplied error string is mapped to a new `ReliabilityError.rePersistenceError` (logged once at the boundary via `reliabilityErr`) and threaded up through every persistence-touching proc to the public API, where the caller decides what to do. Request-driven paths (wrap/unwrap/markDependenciesMet/ensureChannel/removeChannel/reset) propagate the error; background maintenance loops (periodicBufferSweep, periodicRepairSweep) log and retry on the next tick, since they have no synchronous caller. Tests: in-memory backend gains a `failingOps` injection hook; new "Persistence: error propagation" suite asserts read/write/drop failures surface as `rePersistenceError`. Full suite passes (90 OK). BREAKING CHANGE: the `Persistence` contract signature changed; custom backends must return `Result` and `ok()` on success. Bumped to 0.3.0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(persistence): add snapshot types and codec (phase 0) Introduce atomic-snapshot persistence types that will replace the current fine-grained 13-proc Persistence interface. This commit is purely additive: no existing call site changes, no behaviour change. New types (sds/types/): - channel_meta.nim — ChannelMeta (atomic per-channel snapshot blob), ChannelData (bootstrap payload), OutgoingRepairKV / IncomingRepairKV (flattened map entries for protobuf wire shape). - history_update.nim — HistoryUpdate (combined append/evict payload for the message log). New codec (sds/snapshot_codec.nim): - Protobuf encode/decode for all new types, reusing the existing SdsMessage and HistoryEntry encoders from sds/protobuf.nim. - Explicit schemaVersion=1 on ChannelMeta; decoder rejects unknown versions loudly rather than silently truncating. - Time encoded as int64 unix milliseconds. Tests (tests/test_snapshot_codec.nim): - 13 round-trip cases covering empty, single-entry, full-buffer, and repair-heavy snapshots; ChannelData ordering; HistoryUpdate variants; schemaVersion rejection. Planning artefacts: - ANALYSIS_SDS_PERSISTENCE.md — problem statement (partial-write divergence, chatty call rate, non-fatal-error policy gap). - ANALYSIS_SNAPSHOT_SAVE_POINTS.md — exact save points per protocol op and projected call rates. - PLAN_SNAPSHOT_PERSISTENCE.md — phased refactor plan; this commit implements phase 0. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(persistence): add PersistenceV2 interface alongside legacy (phase 1) Introduce the 5-proc snapshot-based Persistence interface that will replace the legacy 13-proc one. Both coexist on `ReliabilityManager` so phase 2 can migrate protocol ops one at a time without breaking existing callers. New file: - sds/types/persistence_v2.nim — `PersistenceV2` type with saveChannelMeta / updateHistory / loadChannel / dropChannel / setRetrievalHint. `noOpPersistenceV2()` default. Doc-comments capture the atomicity pairing (meta save + history update issued back-to-back under the channel lock) and the non-fatal failure policy from PLAN §8. Modified: - sds/types/reliability_manager.nim — adds `persistenceV2: PersistenceV2` field alongside `persistence`; constructor takes both, both default to no-op. - sds.nim — `newReliabilityManager` plumbs the new optional parameter. - AGENTS.md / CLAUDE.md — GitNexus index re-indexed after phase 0 + phase 1 additions; symbol counts updated by `npx gitnexus analyze`. No call site uses the new interface yet — that's phase 2. All existing tests still pass against the legacy interface. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(persistence): migrate runRepairSweep to PersistenceV2 (phase 2.1) Per-entry removeIncomingRepair / removeOutgoingRepair calls are replaced by a single trySaveMeta per *dirty* channel at the end of that channel's sweep. Failure is logged but does NOT abort the sweep — in-memory state is the source of truth (PLAN_SNAPSHOT_PERSISTENCE.md §8). Helpers added in sds/sds_utils.nim: - snapshotMeta(channel) — capture current ChannelContext as ChannelMeta blob (flattens Table-keyed buffers to seqs for the wire shape). - trySaveMeta(rm, channelId, channel) — best-effort meta snapshot save; logs on failure, never propagates. - tryUpdateHistory(rm, channelId, append, evict) — best-effort history update; skips the call entirely when both lists are empty (HistoryUpdate contract). Call-rate impact for runRepairSweep: - Before: N persistence calls per expired entry per channel. - After: at most 1 saveChannelMeta per dirty channel; 0 on idle channels (matches the dirty-flag floor in ANALYSIS_SNAPSHOT_SAVE_POINTS). All existing tests pass — including the 3 SDS-R Repair Sweep tests that directly exercise this proc. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(persistence): migrate checkUnacknowledgedMessages to PersistenceV2 (phase 2.2) Per-entry saveOutgoing / removeOutgoing calls are replaced by one trySaveMeta at the end of the pass, conditional on a dirty flag (resend attempt incremented, or entry expired). Pass succeeds even if the save fails — next tick reissues the snapshot. Call-rate impact: - Before: N persistence calls per affected entry per pass. - After: at most 1 saveChannelMeta per pass; 0 when nothing aged out. All existing tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(persistence): add V2 meta snapshot saves to foreground ops (phase 2A) Wires `trySaveMeta` into the three public protocol ops that mutate per-channel state — wrapOutgoingMessage, unwrapReceivedMessage, and markDependenciesMet — at the operation's end, under the channel lock. Legacy fine-grained persistence calls REMAIN in place; this commit is additive. Both interfaces persist the same state simultaneously, so all existing tests pass and a real backend wired to either interface continues to work. Phase 2B will strip the legacy calls. Save points match the §"Save Points" table in ANALYSIS_SNAPSHOT_SAVE_POINTS.md exactly: - wrapOutgoingMessage: 1 save (always) - unwrapReceivedMessage: 1 save on every path including duplicate (the duplicate path still mutates the repair buffers) - markDependenciesMet: 1 save after the processIncomingBuffer cascade Non-fatal failure policy (PLAN §8): trySaveMeta logs and continues; the protocol op never returns rePersistenceError for snapshot failures. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(persistence): strip legacy interface from protocol path; migrate tests to V2 (phase 2B+2C+2D) End-state of phase 2: the protocol code no longer issues any legacy fine-grained Persistence calls. All state survives via the snapshot-based PersistenceV2 interface — one trySaveMeta per op end, plus tryUpdateHistory batched inside addToHistory. The legacy Persistence field on ReliabilityManager remains for backwards compatibility; phase 3 deletes it. Protocol changes (sds.nim, sds/sds_utils.nim): - reviewAckStatus, processIncomingBuffer, updateLamportTimestamp → pure in-memory; no per-mutation persistence. - addToHistory: replaces appendLogEntry+removeLogEntry with a single tryUpdateHistory call carrying (append, evict) atomically. - getRecentHistoryEntries: setRetrievalHint switched to V2; non-fatal. - wrapOutgoingMessage, unwrapReceivedMessage, markDependenciesMet: all per-row saveOutgoing / removeOutgoing / saveIncoming / removeIncoming / saveOutgoingRepair / removeOutgoingRepair / saveIncomingRepair / removeIncomingRepair calls removed (16 call sites in total). State is captured by the op-end trySaveMeta added in phase 2A. - getOrCreateChannel: bootstraps from persistenceV2.loadChannel. - dropChannelFromPersistence: uses persistenceV2.dropChannel. Failure policy (PLAN_SNAPSHOT_PERSISTENCE.md §8): - Foreground ops (wrap, unwrap, markDeps, sweeps): non-fatal — trySaveMeta / tryUpdateHistory log and continue; the protocol op returns ok regardless of disk failure. In-memory state is the source of truth; the next op re-issues a complete snapshot and disk catches up automatically. - Durability-intent ops (removeChannel, resetReliabilityManager via dropChannelFromPersistence; getOrCreateChannel via loadChannel): still propagate rePersistenceError, because the caller asked us to confirm a disk operation and we cannot silently lie. Test infrastructure: - tests/in_memory_persistence_v2.nim: new V2 adapter mock that decomposes the meta blob into the existing InMemoryStore shape so test assertions on store.outgoing / store.incoming / etc. continue to work without change. - tests/test_persistence.nim: 17 tests, all rewritten against V2. - 13 state-survival tests carry over with identical assertions. - "loadChannel failure surfaces as err on bootstrap" — bootstrap keeps durability-intent semantics. - "saveChannelMeta failure during send does NOT surface" — deliberate inversion of the legacy "write failure surfaces as err" test. Asserts the new non-fatal policy: op returns ok, in-memory state correct, disk re-syncs on the next op. - "updateHistory failure during send does NOT surface" — same policy applied to the history path. - "dropChannel failure during removeChannel surfaces as err" — kept. - All 17 tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(persistence): delete legacy interface; rename PersistenceV2 -> Persistence (phase 3) End-state of the snapshot-persistence refactor. The legacy 13-proc Persistence interface and its noOpPersistence are gone; the 5-proc snapshot-based interface (formerly PersistenceV2) takes their place under the canonical name. Source: - sds/types/persistence.nim: replaced 13-proc contract with the 5-proc snapshot interface (saveChannelMeta, updateHistory, loadChannel, dropChannel, setRetrievalHint). noOpPersistence returns ok everywhere and an empty ChannelData on load. - sds/types/persistence_v2.nim: removed. - sds/types/reliability_manager.nim: dropped the second persistenceV2 field; constructor takes a single `persistence: Persistence`. - sds/sds_utils.nim: rm.persistenceV2.X -> rm.persistence.X; doc-comments updated. - sds.nim: dropped the persistenceV2 parameter from newReliabilityManager. Tests: - tests/in_memory_persistence_v2.nim: removed; its content moved to... - tests/in_memory_persistence.nim: replaces the old legacy mock with the snapshot adapter under the canonical filename. Same InMemoryStore shape so test assertions stay unchanged. - tests/test_persistence.nim: ctor param renamed, suite name de-prefixed. FFI smoke (`nimble libsdsDynamicMac`, refc/threads:on): builds clean. All 4 test suites pass: - test_bloom - test_reliability - test_persistence (17 V2 tests) - test_snapshot_codec (13 codec round-trip tests) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Persisting persistence redesign plan for reference * refactor(persistence): R2 pending-write queue + per-op accumulator (PR #72 review fix) Addresses all three substantive review findings on PR #72 in one structural change: fold the per-op accumulator and the R2 retry buffer into a single queue on `ChannelContext`, flushed once at op end. Changes: - sds/types/channel_context.nim: add `pendingHistoryAppends` (`OrderedSet[SdsMessageID]`) and `pendingHistoryEvicts` (`HashSet[SdsMessageID]`) fields. Only ids are stored — the full SdsMessage is looked up from `messageHistory` at flush time. Documented invariant: every id in pendingHistoryAppends is also in messageHistory, upheld by the merge rule. - sds/sds_utils.nim: * `queueHistoryAppend(channel, msgId)` / `queueHistoryEvict(channel, msgId)` — "latest-wins" merge: append cancels any pending evict and vice versa. Symmetric, simple, handles the evict-then-re-add sequence correctly (SDS-R repair re-delivering an evicted message while the backend is unreachable). * `tryUpdateHistory(rm, channelId)` — no more list params; flushes the channel's pending queue. Dual role: per-op accumulator (multiple `addToHistory` calls within one op queue together and flush as one round-trip) AND R2 retry buffer (a failed flush leaves the queue populated for the next op to retry). * `addToHistory` queues via the helpers; does not call persistence. * Pending queue cleared on `cleanup` and `removeChannel`. - sds.nim: * `processIncomingBuffer` returns to its single-arg signature — the queue lives on the channel, no parameter threading needed. * `wrapOutgoingMessage`, `unwrapReceivedMessage` (all three paths), `markDependenciesMet` issue exactly one `trySaveMeta` + `tryUpdateHistory` pair at op end, under the lock, with no intervening `await`-of-other-work. Matches the Persistence atomicity contract documented in `sds/types/persistence.nim`. * Pending queue cleared in `resetReliabilityManager`. - tests/test_persistence.nim: * Direct `addToHistory` callers (state-survival setup) now follow with explicit `tryUpdateHistory(channelId)` to flush. Reflects the production op-end flush pattern. * New: `updateHistory failure is retried via R2 pending-write queue` — verifies that two failed sends leave both messages on the queue, and a third successful send drains the whole queue in one call. * New: `pending queue survives idle ops` — verifies that an op with no history changes of its own still flushes a previously-failed batch at op end. * New: `evict-then-re-add merge rule preserves the re-added message on disk` — regression for the "latest-wins" merge rule. The original "evict-wins" rule would silently drop the re-add and leave the message permanently absent from disk; this test would fail under that rule and passes under the corrected one. Resolves PR #72 review comments: - #1 (delta loss on failed updateHistory) — R2 retry queue. - #2 (cascade chattiness — N updateHistory calls per op) — queue collects cascaded entries, flushed as one batch. - #3 (atomicity contract mismatch) — implementation now matches the documented "saveChannelMeta then updateHistory back-to-back" pairing. Test summary: 50 tests pass (47 prior + 3 new R2/merge-rule tests). FFI dylib (`nimble libsdsDynamicMac`, refc + threads:on): clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-01 12:24:38 +02:00
## Round-trip tests for the snapshot persistence codec.
## Each `encode` → `decode` cycle must preserve every field exactly.
import std/[times, sets, unittest]
import results
import ../sds/snapshot_codec
import
../sds/types/[
sds_message, sds_message_id, history_entry, unacknowledged_message,
incoming_message, repair_entry,
]
converter toParticipantID(s: string): SdsParticipantID =
s.SdsParticipantID
proc mkMsg(id: string, ts: int64 = 1, content: seq[byte] = @[]): SdsMessage =
SdsMessage.init(
messageId = id,
lamportTimestamp = ts,
causalHistory = @[],
channelId = "chan",
content = content,
bloomFilter = @[],
senderId = "alice",
repairRequest = @[],
)
proc mkHistEntry(id: string): HistoryEntry =
HistoryEntry.init(messageId = id, senderId = "alice")
suite "snapshot codec — ChannelMeta":
test "empty meta round-trips":
let m = ChannelMeta.init()
let buf = encode(m).buffer
let dec = ChannelMeta.decode(buf).get()
check:
dec.schemaVersion == ChannelMetaSchemaVersion
dec.lamportTimestamp == 0
dec.outgoingBuffer.len == 0
dec.incomingBuffer.len == 0
dec.outgoingRepairBuffer.len == 0
dec.incomingRepairBuffer.len == 0
test "meta with lamport and single outgoing entry":
var m = ChannelMeta.init()
m.lamportTimestamp = 42
m.outgoingBuffer.add(
UnacknowledgedMessage.init(
message = mkMsg("m1", 42, @[1.byte, 2, 3]),
sendTime = fromUnix(1_700_000_000),
resendAttempts = 2,
)
)
let buf = encode(m).buffer
let dec = ChannelMeta.decode(buf).get()
check:
dec.lamportTimestamp == 42
dec.outgoingBuffer.len == 1
dec.outgoingBuffer[0].message.messageId == "m1"
dec.outgoingBuffer[0].message.content == @[1.byte, 2, 3]
dec.outgoingBuffer[0].resendAttempts == 2
dec.outgoingBuffer[0].sendTime.toUnix == 1_700_000_000
test "meta with incoming entry carrying missing deps":
var m = ChannelMeta.init()
var deps = initHashSet[SdsMessageID]()
deps.incl("dep1")
deps.incl("dep2")
m.incomingBuffer.add(
IncomingMessage.init(message = mkMsg("m2"), missingDeps = deps)
)
let buf = encode(m).buffer
let dec = ChannelMeta.decode(buf).get()
check:
dec.incomingBuffer.len == 1
dec.incomingBuffer[0].message.messageId == "m2"
dec.incomingBuffer[0].missingDeps == deps
test "meta with both repair buffers populated":
var m = ChannelMeta.init()
m.outgoingRepairBuffer.add(
OutgoingRepairKV(
messageId: "missing1",
entry: OutgoingRepairEntry.init(
outHistEntry = mkHistEntry("missing1"),
minTimeRepairReq = fromUnix(1_700_000_100),
),
)
)
m.incomingRepairBuffer.add(
IncomingRepairKV(
messageId: "requested1",
entry: IncomingRepairEntry.init(
inHistEntry = mkHistEntry("requested1"),
cachedMessage = @[9.byte, 8, 7, 6],
minTimeRepairResp = fromUnix(1_700_000_200),
),
)
)
let buf = encode(m).buffer
let dec = ChannelMeta.decode(buf).get()
check:
dec.outgoingRepairBuffer.len == 1
dec.outgoingRepairBuffer[0].messageId == "missing1"
dec.outgoingRepairBuffer[0].entry.minTimeRepairReq.toUnix ==
1_700_000_100
dec.incomingRepairBuffer.len == 1
dec.incomingRepairBuffer[0].messageId == "requested1"
dec.incomingRepairBuffer[0].entry.cachedMessage == @[9.byte, 8, 7, 6]
dec.incomingRepairBuffer[0].entry.minTimeRepairResp.toUnix ==
1_700_000_200
test "fully-populated meta — multiple entries each buffer":
var m = ChannelMeta.init()
m.lamportTimestamp = 999
for i in 0 ..< 5:
m.outgoingBuffer.add(
UnacknowledgedMessage.init(
message = mkMsg("o" & $i, int64(i), @[byte(i)]),
sendTime = fromUnix(1_700_000_000 + i.int64),
resendAttempts = i,
)
)
for i in 0 ..< 3:
var deps = initHashSet[SdsMessageID]()
deps.incl("dep" & $i)
m.incomingBuffer.add(
IncomingMessage.init(
message = mkMsg("i" & $i, int64(100 + i)), missingDeps = deps
)
)
for i in 0 ..< 4:
m.outgoingRepairBuffer.add(
OutgoingRepairKV(
messageId: "or" & $i,
entry: OutgoingRepairEntry.init(
outHistEntry = mkHistEntry("or" & $i),
minTimeRepairReq = fromUnix(1_700_000_300 + i.int64),
),
)
)
for i in 0 ..< 2:
m.incomingRepairBuffer.add(
IncomingRepairKV(
messageId: "ir" & $i,
entry: IncomingRepairEntry.init(
inHistEntry = mkHistEntry("ir" & $i),
cachedMessage = @[byte(i), byte(i + 1)],
minTimeRepairResp = fromUnix(1_700_000_400 + i.int64),
),
)
)
let buf = encode(m).buffer
let dec = ChannelMeta.decode(buf).get()
check:
dec.lamportTimestamp == 999
dec.outgoingBuffer.len == 5
dec.incomingBuffer.len == 3
dec.outgoingRepairBuffer.len == 4
dec.incomingRepairBuffer.len == 2
dec.outgoingBuffer[4].message.messageId == "o4"
dec.outgoingBuffer[4].resendAttempts == 4
dec.outgoingRepairBuffer[3].messageId == "or3"
dec.incomingRepairBuffer[1].entry.cachedMessage == @[1.byte, 2]
test "decoder rejects unknown schemaVersion":
var m = ChannelMeta.init()
m.schemaVersion = 999'u32
let buf = encode(m).buffer
check ChannelMeta.decode(buf).isErr
suite "snapshot codec — ChannelData":
test "empty channel data round-trips":
let d = ChannelData.init()
let buf = encode(d).buffer
let dec = ChannelData.decode(buf).get()
check:
dec.meta.schemaVersion == ChannelMetaSchemaVersion
dec.messageHistory.len == 0
test "channel data with meta and history preserves order":
var d = ChannelData.init()
d.meta.lamportTimestamp = 17
d.messageHistory.add(mkMsg("h1", 1))
d.messageHistory.add(mkMsg("h2", 2))
d.messageHistory.add(mkMsg("h3", 3))
let buf = encode(d).buffer
let dec = ChannelData.decode(buf).get()
check:
dec.meta.lamportTimestamp == 17
dec.messageHistory.len == 3
dec.messageHistory[0].messageId == "h1"
dec.messageHistory[1].messageId == "h2"
dec.messageHistory[2].messageId == "h3"
suite "snapshot codec — HistoryUpdate":
test "empty update reports isEmpty (callers skip persistence)":
# By contract (HistoryUpdate doc): when both append and evict are
# empty, callers MUST skip the persistence call entirely. The codec
# is not required to round-trip an empty update — minprotobuf's
# finish() refuses an empty buffer, by design.
let u = HistoryUpdate.init()
check u.isEmpty
test "append-only update":
var u = HistoryUpdate.init()
u.append.add(mkMsg("a1"))
u.append.add(mkMsg("a2"))
let buf = encode(u).buffer
let dec = HistoryUpdate.decode(buf).get()
check:
dec.append.len == 2
dec.append[0].messageId == "a1"
dec.append[1].messageId == "a2"
dec.evict.len == 0
test "evict-only update":
var u = HistoryUpdate.init()
u.evict = @["e1", "e2", "e3"]
let buf = encode(u).buffer
let dec = HistoryUpdate.decode(buf).get()
check:
dec.append.len == 0
dec.evict == @["e1", "e2", "e3"]
test "mixed append + evict update":
var u = HistoryUpdate.init()
u.append.add(mkMsg("new"))
u.evict = @["old1", "old2"]
let buf = encode(u).buffer
let dec = HistoryUpdate.decode(buf).get()
check:
dec.append.len == 1
dec.append[0].messageId == "new"
dec.evict == @["old1", "old2"]