mirror of
https://github.com/logos-messaging/nim-sds.git
synced 2026-07-02 13:59:41 +00:00
abc36f61aa
Addresses all three substantive review findings on PR #72 in one structural change: fold the per-op accumulator and the R2 retry buffer into a single queue on `ChannelContext`, flushed once at op end. Changes: - sds/types/channel_context.nim: add `pendingHistoryAppends` (`OrderedSet[SdsMessageID]`) and `pendingHistoryEvicts` (`HashSet[SdsMessageID]`) fields. Only ids are stored — the full SdsMessage is looked up from `messageHistory` at flush time. Documented invariant: every id in pendingHistoryAppends is also in messageHistory, upheld by the merge rule. - sds/sds_utils.nim: * `queueHistoryAppend(channel, msgId)` / `queueHistoryEvict(channel, msgId)` — "latest-wins" merge: append cancels any pending evict and vice versa. Symmetric, simple, handles the evict-then-re-add sequence correctly (SDS-R repair re-delivering an evicted message while the backend is unreachable). * `tryUpdateHistory(rm, channelId)` — no more list params; flushes the channel's pending queue. Dual role: per-op accumulator (multiple `addToHistory` calls within one op queue together and flush as one round-trip) AND R2 retry buffer (a failed flush leaves the queue populated for the next op to retry). * `addToHistory` queues via the helpers; does not call persistence. * Pending queue cleared on `cleanup` and `removeChannel`. - sds.nim: * `processIncomingBuffer` returns to its single-arg signature — the queue lives on the channel, no parameter threading needed. * `wrapOutgoingMessage`, `unwrapReceivedMessage` (all three paths), `markDependenciesMet` issue exactly one `trySaveMeta` + `tryUpdateHistory` pair at op end, under the lock, with no intervening `await`-of-other-work. Matches the Persistence atomicity contract documented in `sds/types/persistence.nim`. * Pending queue cleared in `resetReliabilityManager`. - tests/test_persistence.nim: * Direct `addToHistory` callers (state-survival setup) now follow with explicit `tryUpdateHistory(channelId)` to flush. Reflects the production op-end flush pattern. * New: `updateHistory failure is retried via R2 pending-write queue` — verifies that two failed sends leave both messages on the queue, and a third successful send drains the whole queue in one call. * New: `pending queue survives idle ops` — verifies that an op with no history changes of its own still flushes a previously-failed batch at op end. * New: `evict-then-re-add merge rule preserves the re-added message on disk` — regression for the "latest-wins" merge rule. The original "evict-wins" rule would silently drop the re-add and leave the message permanently absent from disk; this test would fail under that rule and passes under the corrected one. Resolves PR #72 review comments: - #1 (delta loss on failed updateHistory) — R2 retry queue. - #2 (cascade chattiness — N updateHistory calls per op) — queue collects cascaded entries, flushed as one batch. - #3 (atomicity contract mismatch) — implementation now matches the documented "saveChannelMeta then updateHistory back-to-back" pairing. Test summary: 50 tests pass (47 prior + 3 new R2/merge-rule tests). FFI dylib (`nimble libsdsDynamicMac`, refc + threads:on): clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
nim-sds
Nim implementation of the e2e reliability protocol.
Prerequisites
- Nix package manager
Quick start
git clone https://github.com/logos-messaging/nim-sds.git
cd nim-sds
# Build the shared library
nix build '.#libsds'
# Run tests
nix develop --command nimble test
Building
Desktop
nix build --print-out-paths '.#libsds'
Android
nix build --print-out-paths '.#libsds-android-arm64'
nix build --print-out-paths '.#libsds-android-amd64'
nix build --print-out-paths '.#libsds-android-x86'
nix build --print-out-paths '.#libsds-android-arm'
iOS
nix build --print-out-paths '.#libsds-ios'
Development shell
Enter the dev shell:
nix develop
Build using nimble tasks:
# Dynamic library (auto-detects OS)
nimble libsdsDynamicMac # macOS
nimble libsdsDynamicLinux # Linux
nimble libsdsDynamicWindows # Windows
# Static library
nimble libsdsStaticMac # macOS
nimble libsdsStaticLinux # Linux
nimble libsdsStaticWindows # Windows
Run tests:
nimble test
The built library is output to build/.
Android (without Nix)
Download the latest Android NDK:
cd ~
wget https://dl.google.com/android/repository/android-ndk-r27c-linux.zip
unzip android-ndk-r27c-linux.zip
Add to ~/.bashrc:
export ANDROID_NDK_ROOT=$HOME/android-ndk-r27c
export PATH=$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/bin:$PATH
Then build:
ARCH=arm64 nimble libsdsAndroid
| Architecture | Command |
|---|---|
| arm64 | ARCH=arm64 nimble libsdsAndroid |
| amd64 | ARCH=amd64 nimble libsdsAndroid |
| x86 | ARCH=x86 nimble libsdsAndroid |
The library is output to build/libsds.so.
Dependency management
Dependencies are managed by Nimble and pinned via nimble.lock.
To set up dependencies locally:
nimble setup -l
To update dependencies:
nimble lock
After updating nimble.lock, the Nix outputHash in nix/deps.nix must be recalculated
by running nix build and updating the hash from the error output.
License
Licensed under either of Apache License, Version 2.0 or MIT license at your option.