**nim-sds** is a Nim implementation of the **Scalable Data Sync (SDS)** protocol ([spec](https://lip.logos.co/ift-ts/raw/sds.html), IFT LIP-109). SDS achieves end-to-end reliability when consolidating distributed logs in a decentralized manner — participants broadcast messages over a P2P transport, maintain per-channel append-only logs, and use causal ordering to reach consistent state across all nodes.
The library exposes its functionality via a C-compatible FFI so it can be embedded in applications on any platform. Go bindings are maintained in a separate repo: [logos-messaging/sds-go-bindings](https://github.com/logos-messaging/sds-go-bindings).
---
## Protocol Concepts
Understanding these is essential before modifying any core code.
### Message format
Each SDS message carries (`sds/message.nim`, `sds/protobuf.nim`):
| `lamport_timestamp` | Logical clock for ordering |
| `causal_history` | IDs of the 2–3 most recent messages the sender has seen (dependencies) |
| `bloom_filter` | Compact summary of all message IDs the sender has received |
| `content` | Application payload |
### Sending a message (`sds/sds_utils.nim` → `wrapOutgoingMessage`)
1. Increment the per-channel Lamport timestamp to `max(current_time_ms, timestamp + 1)`.
2. Attach causal history from the local log tail.
3. Embed the current bloom filter snapshot.
### Receiving a message (`sds/sds_utils.nim` → `unwrapReceivedMessage`)
1. Deduplicate by `message_id`.
2. Check causal dependencies — if any predecessor is missing, buffer the message.
3. When all dependencies are met, deliver: insert into the ordered local log (Lamport timestamp, tie-break by ascending `message_id`).
4. Record `message_id` in the bloom filter.
### Periodic sync
A node periodically broadcasts a message with empty content carrying an updated Lamport timestamp and bloom filter. These sync messages are not persisted and are excluded from causal chains. They help peers detect gaps in their logs.
### SDS-R (Repair extension)
Defined in the spec but **not yet implemented** in this library.
Used to compactly summarise which messages a node has received, so peers can identify gaps without exchanging full ID lists. The rolling variant automatically resets when capacity is exceeded.
- **Backward compat**: `sds/protobuf.nim` supports old and new causal history formats — do not remove the legacy decode path
---
## Building
**Nimble** is the primary build tool. Desktop library targets:
```bash
nimble libsdsDynamicMac # macOS .dylib
nimble libsdsDynamicLinux # Linux .so
nimble libsdsStaticMac # macOS .a
nimble libsdsStaticLinux # Linux .a
```
**Nix** (`flake.nix`) and **Make** (`Makefile`) are optional conveniences that wrap Nimble for reproducible and cross-platform (including Android/iOS) builds.
## Dependency Management
Nimble dependencies are locked in `nimble.lock`.
```bash
nimble setup -l # local setup
nimble lock # update lock after changing sds.nimble
```
If using Nix, also recalculate the fixed-output hash in `nix/deps.nix` after updating `nimble.lock` (run `nix build`, copy the expected hash from the error, paste into `outputHash`).
This project is indexed by GitNexus as **nim-sds** (889 symbols, 1437 relationships, 45 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
## Always Do
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
## Never Do
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
## Resources
| Resource | Use for |
|----------|---------|
| `gitnexus://repo/nim-sds/context` | Codebase overview, check index freshness |
| `gitnexus://repo/nim-sds/clusters` | All functional areas |
| `gitnexus://repo/nim-sds/processes` | All execution flows |
