logos-messaging/specs
1
0
Fork 0
mirror of https://github.com/logos-messaging/specs.git synced 2026-05-28 15:59:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: transport-agnostic

Browse Source
This commit is contained in:
Igor Sirotin 2026-04-28 21:32:42 +01:00
parent b3e52e73b6
commit 09133e0a3f
No known key found for this signature in database
GPG Key ID: 0EABBCB40CB9AD4A
1 changed files with 8 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
standards/application/segmentation.md
View File

@ -8,8 +8,8 @@ status: draft
## Abstract ## Abstract
This specification defines an application-layer protocol for **segmentation** and **reconstruction** of messages carried over a message transport/delivery services with size limitation, when the original payload exceeds said limitation. This specification defines an application-layer protocol for **segmentation** and **reconstruction** of messages carried over a transport/delivery service with a message-size limitation, when the original payload exceeds said limitation.
Applications partition the payload into multiple wire-messages envelopes and reconstruct the original on receipt, Applications partition the payload into multiple transport messages and reconstruct the original on receipt,
even when segments arrive out of order or up to a **predefined percentage** of segments are lost. even when segments arrive out of order or up to a **predefined percentage** of segments are lost.
The protocol uses **Reed–Solomon** erasure coding for fault tolerance. The protocol uses **Reed–Solomon** erasure coding for fault tolerance.
All messages are wrapped in a `SegmentMessageProto`, including those that fit in a single segment. All messages are wrapped in a `SegmentMessageProto`, including those that fit in a single segment.
@ -17,7 +17,8 @@ Implementations **MAY** opt into a [backwards-compatible mode](#backwards-compat
## Motivation ## Motivation
Waku Relay deployments typically propagate envelopes up to **150 KB** as per [64/WAKU2-NETWORK - Message](https://rfc.vac.dev/waku/standards/core/64/network#message-size). Many message transport and delivery protocols impose a maximum message size that restricts the size of application payloads.
For example, Waku Relay typically propagates messages up to **150 KB** as per [64/WAKU2-NETWORK - Message](https://rfc.vac.dev/waku/standards/core/64/network#message-size).
To support larger application payloads, a segmentation layer is required. To support larger application payloads, a segmentation layer is required.
This specification enables larger messages by partitioning them into multiple envelopes and reconstructing them at the receiver. This specification enables larger messages by partitioning them into multiple envelopes and reconstructing them at the receiver.
Erasure-coded parity segments provide resilience against partial loss or reordering. Erasure-coded parity segments provide resilience against partial loss or reordering.
@ -95,7 +96,7 @@ To transmit a payload, the sender:
- The `entire_message_hash` - The `entire_message_hash`
- Either data-segment indices (`segments_count`, `index`) or parity-segment indices (`parity_segments_count`, `parity_segment_index`) - Either data-segment indices (`segments_count`, `index`) or parity-segment indices (`parity_segments_count`, `parity_segment_index`)
- The raw payload data - The raw payload data
- Send all segments as individual Waku envelopes, - Send each segment as an individual transport message according to the underlying transport protocol,
preserving application-level metadata (e.g., content topic). preserving application-level metadata (e.g., content topic).
This yields a deterministic wire format: every transmitted payload is a `SegmentMessageProto`. Implementations introducing segmentation into a deployment with peers that predate this specification **MAY** instead operate in [backwards-compatible mode](#backwards-compatibility). This yields a deterministic wire format: every transmitted payload is a `SegmentMessageProto`. Implementations introducing segmentation into a deployment with peers that predate this specification **MAY** instead operate in [backwards-compatible mode](#backwards-compatibility).
@ -177,7 +178,7 @@ Libraries **SHOULD** require only `segmentSize` from the application for normal
- **Language / Package:** Nim; - **Language / Package:** Nim;
**Nimble** package manager **Nimble** package manager
- **Intended for:** all Waku nodes at the application layer - **Intended for:** application-layer use over any transport with message-size constraints
--- ---
@ -199,7 +200,7 @@ To mitigate resource exhaustion:
- Limit concurrent reconstructions and per-sender storage - Limit concurrent reconstructions and per-sender storage
- Enforce timeouts and size caps - Enforce timeouts and size caps
- Validate segment counts (≤ 256) - Validate segment counts (≤ 256)
- Consider rate-limiting using [17/WAKU2-RLN-RELAY](https://rfc.vac.dev/waku/standards/core/17/rln-relay) - Consider rate-limiting at the transport layer (for example, via [17/WAKU2-RLN-RELAY](https://rfc.vac.dev/waku/standards/core/17/rln-relay) on Waku)
### Compatibility ### Compatibility
@ -216,7 +217,7 @@ Nodes that do **not** implement this specification cannot reconstruct large mess
**Network impact:** **Network impact:**
- Larger messages increase gossip traffic and storage; - Larger messages increase transport traffic and storage;
operators **SHOULD** consider policy limits operators **SHOULD** consider policy limits
--- ---