diff --git a/.gitignore b/.gitignore index 9a9c03f..1f21592 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,7 @@ tests/* !*.nim ratelimit/* !*.nim +!*.md !*.proto nimble.develop nimble.paths diff --git a/README.md b/README.md index 345ce8a..2c41eb9 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,11 @@ make build-go # Verify Go bindings compile make run-go-example ``` +## Modules + +[Rate limit](ratelimit/README.md) +[Segmentation](ratelimit/README.segmentation.md) + ## TODO - [ ] Follow closer the integrations from [this repo](https://github.com/logos-co/nim-c-library-guide) diff --git a/README.segmentation.md b/README.segmentation.md new file mode 100644 index 0000000..7d7edb5 --- /dev/null +++ b/README.segmentation.md @@ -0,0 +1,32 @@ +# Message Segmentation + +Split large payloads into Waku-sized segments with optional Reed–Solomon parity for robust reconstruction. Inbound segmented messages are reassembled and persisted until complete. + +## Features + +- **Segmentation**: Outbound payloads over the configured size are split into segments; smaller payloads pass through unchanged. +- **Reconstruction**: Inbound segments can be processed out of order and reassembled into the original message. +- **Erasure coding**: Adds parity segments at 12.5% of data segments (≤ SegmentsReedsolomonMaxCount), enabling recovery with ~87.5% of segments. +- **Persistence**: Segments and completion state are stored (SQLite) and survive restarts. +- **API simplicity**: Main parameter is the maximum segment size; inject your persistence object. + +## Essential API + +- **Types**: `SegmentationHander`, `Chunk`, `Message` +- **Outbound**: `segmentMessage(handler, chunk: Chunk): Result[seq[Chunk], string]` +- **Inbound**: `handleSegmentationLayer(handler, message: var Message): Result[void, string]` +- **Helpers**: `isParityMessage(segment: SegmentMessage): bool` +- **Parameters**: `segmentSize` (bytes); provide `SegmentationPersistence` instance + +Notes +- Parity rate is 12.5% (`SegmentsParityRate = 0.125`). + +## Wire format + +Segments are protobuf-encoded (`segment_message.proto`): + +## Tests + +See `tests/test_segmentation.nim` for examples covering in-order/out-of-order and parity recovery paths. + + diff --git a/ratelimit/README.md b/ratelimit/README.md new file mode 100644 index 0000000..30494cb --- /dev/null +++ b/ratelimit/README.md @@ -0,0 +1,45 @@ +# Rate-Limited Message Manager (Nim) + +Rate limiter for sending messages to a provided delivery service. Implements a token-bucket per epoch, queues by priority, and persists queue and quota across restarts. Intended to provide a good UX for sending messages in an RLN-protected network, so that developers can queue, prioritize and inform end users when quotas might be close to being consumed. + +## Features + +- **Rate limit per epoch**: Fixed-capacity token bucket (messages per epoch, RLN-style window). +- **Priorities**: **Critical**, **Normal**, **Optional**. +- **Low/empty quota behavior**: + - Low quota: Critical sent, Normal queued, Optional dropped. + - Exhausted: Critical queued, Normal queued, Optional dropped. +- **Batch all-or-none**: Pass messages as a batch; either the whole batch is sent, queued, or dropped. +- **Status tracking**: `PassedToSender`, `Enqueued`, `Dropped`, `DroppedBatchTooLarge`, `DroppedFailedToEnqueue`. +- **Persistence**: Queues and quota state survive restarts (SQLite). + +## Install + +- Built as part of this Nimble project. Add this repo as a dependency or develop locally, then import `ratelimit/ratelimit_manager` and `ratelimit/store`. + +## API (essentials) + +- **Create**: `await RateLimitStore[T].new(db)`; `await RateLimitManager[T].new(store, sender, capacity, duration, sleepDuration)` +- **Run**: `await manager.start()` / `await manager.stop()` +- **Send**: `await manager.sendOrEnqueue(@[(msgId, msg), ...], Priority)` +- **Quota**: `manager.getQuota()` → `(remaining, total)` +- **Message Status**: `await manager.getMessageStatus(msgId)` + +Notes: +- Batches larger than 30% of total capacity are dropped to prevent starvation (`DroppedBatchTooLarge`). +- Sender is user-provided. Message outcome is recorded via `MessageStatus`. + +## Persistence + +Backed by SQLite. Migrations create the following tables: +- `kv_store` (stores bucket state) +- `ratelimit_queues` (persisted batches per priority) +- `ratelimit_message_status` (latest status per `msg_id`) + +Apply migrations via `chat_sdk/migration.runMigrations(db)`. + +## Tests + +See `tests/test_ratelimit_manager.nim` for usage and expected behaviors. + +