From ec07ec40966f6482dcbf1dadf6951c50ec079575 Mon Sep 17 00:00:00 2001
From: pablo <p.lopez.lpz@gmail.com>
Date: Sun, 14 Sep 2025 20:02:18 +0300
Subject: [PATCH] feat: readmes for rate limit and segmentation

---
 .gitignore             |  1 +
 README.md              |  5 +++++
 README.segmentation.md | 32 ++++++++++++++++++++++++++++++
 ratelimit/README.md    | 45 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 83 insertions(+)
 create mode 100644 README.segmentation.md
 create mode 100644 ratelimit/README.md

diff --git a/.gitignore b/.gitignore
index 9a9c03f..1f21592 100644
--- a/.gitignore
+++ b/.gitignore
@@ -26,6 +26,7 @@ tests/*
 !*.nim
 ratelimit/*
 !*.nim
+!*.md
 !*.proto
 nimble.develop
 nimble.paths
diff --git a/README.md b/README.md
index 345ce8a..2c41eb9 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,11 @@ make build-go     # Verify Go bindings compile
 make run-go-example
 ```
 
+## Modules
+
+[Rate limit](ratelimit/README.md)
+[Segmentation](ratelimit/README.segmentation.md)
+
 ## TODO
 
 - [ ] Follow closer the integrations from [this repo](https://github.com/logos-co/nim-c-library-guide)
diff --git a/README.segmentation.md b/README.segmentation.md
new file mode 100644
index 0000000..7d7edb5
--- /dev/null
+++ b/README.segmentation.md
@@ -0,0 +1,32 @@
+# Message Segmentation 
+
+Split large payloads into Waku-sized segments with optional Reed–Solomon parity for robust reconstruction. Inbound segmented messages are reassembled and persisted until complete.
+
+## Features
+
+- **Segmentation**: Outbound payloads over the configured size are split into segments; smaller payloads pass through unchanged.
+- **Reconstruction**: Inbound segments can be processed out of order and reassembled into the original message.
+- **Erasure coding**: Adds parity segments at 12.5% of data segments (≤ SegmentsReedsolomonMaxCount), enabling recovery with ~87.5% of segments.
+- **Persistence**: Segments and completion state are stored (SQLite) and survive restarts.
+- **API simplicity**: Main parameter is the maximum segment size; inject your persistence object.
+
+## Essential API
+
+- **Types**: `SegmentationHander`, `Chunk`, `Message`
+- **Outbound**: `segmentMessage(handler, chunk: Chunk): Result[seq[Chunk], string]`
+- **Inbound**: `handleSegmentationLayer(handler, message: var Message): Result[void, string]`
+- **Helpers**: `isParityMessage(segment: SegmentMessage): bool`
+- **Parameters**: `segmentSize` (bytes); provide `SegmentationPersistence` instance
+
+Notes
+- Parity rate is 12.5% (`SegmentsParityRate = 0.125`).
+
+## Wire format
+
+Segments are protobuf-encoded (`segment_message.proto`):
+
+## Tests
+
+See `tests/test_segmentation.nim` for examples covering in-order/out-of-order and parity recovery paths.
+
+
diff --git a/ratelimit/README.md b/ratelimit/README.md
new file mode 100644
index 0000000..30494cb
--- /dev/null
+++ b/ratelimit/README.md
@@ -0,0 +1,45 @@
+# Rate-Limited Message Manager (Nim)
+
+Rate limiter for sending messages to a provided delivery service. Implements a token-bucket per epoch, queues by priority, and persists queue and quota across restarts. Intended to provide a good UX for sending messages in an RLN-protected network, so that developers can queue, prioritize and inform end users when quotas might be close to being consumed.
+
+## Features
+
+- **Rate limit per epoch**: Fixed-capacity token bucket (messages per epoch, RLN-style window).
+- **Priorities**: **Critical**, **Normal**, **Optional**.
+- **Low/empty quota behavior**:
+  - Low quota: Critical sent, Normal queued, Optional dropped.
+  - Exhausted: Critical queued, Normal queued, Optional dropped.
+- **Batch all-or-none**: Pass messages as a batch; either the whole batch is sent, queued, or dropped.
+- **Status tracking**: `PassedToSender`, `Enqueued`, `Dropped`, `DroppedBatchTooLarge`, `DroppedFailedToEnqueue`.
+- **Persistence**: Queues and quota state survive restarts (SQLite).
+
+## Install
+
+- Built as part of this Nimble project. Add this repo as a dependency or develop locally, then import `ratelimit/ratelimit_manager` and `ratelimit/store`.
+
+## API (essentials)
+
+- **Create**: `await RateLimitStore[T].new(db)`; `await RateLimitManager[T].new(store, sender, capacity, duration, sleepDuration)`
+- **Run**: `await manager.start()` / `await manager.stop()`
+- **Send**: `await manager.sendOrEnqueue(@[(msgId, msg), ...], Priority)`
+- **Quota**: `manager.getQuota()` → `(remaining, total)`
+- **Message Status**: `await manager.getMessageStatus(msgId)`
+
+Notes:
+- Batches larger than 30% of total capacity are dropped to prevent starvation (`DroppedBatchTooLarge`).
+- Sender is user-provided. Message outcome is recorded via `MessageStatus`.
+
+## Persistence
+
+Backed by SQLite. Migrations create the following tables:
+- `kv_store` (stores bucket state)
+- `ratelimit_queues` (persisted batches per priority)
+- `ratelimit_message_status` (latest status per `msg_id`)
+
+Apply migrations via `chat_sdk/migration.runMigrations(db)`.
+
+## Tests
+
+See `tests/test_ratelimit_manager.nim` for usage and expected behaviors.
+
+