logos-messaging/specs
1
0
Fork 0
mirror of https://github.com/logos-messaging/specs.git synced 2026-01-02 14:13:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: some clarification on defaults for sharding (#76)

Browse Source 
* docs: some clarification on defaults for sharding

* docs: added some details on scaling

* docs: clarifications based on queries
This commit is contained in:
Hanno Cornelius 2025-07-30 17:59:58 +01:00 committed by GitHub
parent a2f5509dd0
commit bfd08e1b47
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 44 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
standards/core/relay-sharding.md
View File

@ -34,9 +34,13 @@ This document also covers discovery of topic shards.
## Named Sharding
> **_Note:_** As of 2025-07-24 freely named sharding has been deprecated from all Waku implementations.
It is still described here as background for static and automatic sharding.
_Named sharding_ offers apps to freely choose pubsub topic names.
It is RECOMMENDED for App protocols to follow the naming structure detailed in [23/WAKU2-TOPICS](https://github.com/vacp2p/rfc-index/blob/main/waku/informational/23/topics.md).
With named sharding, managing discovery falls into the responsibility of apps.
For this reason it is NOT RECOMMENDED that app protocols and Waku implementations make use of named sharding.
From an app protocol point of view, a subscription to a content topic `waku2/xxx` on a shard named /mesh/v1.1.1/xxx would look like:
@ -44,14 +48,20 @@ From an app protocol point of view, a subscription to a content topic `waku2/xxx
## Static Sharding
_Static sharding_ offers a set of shards with fixed names.
_Static sharding_ is an extension of named sharding that offers a set of shards with _fixed_ names.
Assigning content topics to specific shards is up to app protocols,
but the discovery of these shards is managed by Waku.
This is the RECOMMENDED default format for shards chosen by an app protocol.
Static shards are managed in shard clusters of 1024 shards per cluster.
Waku static sharding can manage $2^{16}$ shard clusters.
Each shard cluster is identified by its index (between $0$ and $2^{16}-1$).
It is RECOMMENDED that, for simplification of configuration and various APIs,
all app-level protocols only interact with `cluster` and `shard`
and never the fully-formed pubsub topic,
which is a concern internal to the Waku implementation.
A specific shard cluster is either globally available to all apps,
specific for an app protocol,
or reserved for automatic sharding (see next section).
@ -74,6 +84,9 @@ The global shard with index 0 and the "all app protocols" range are treated in t
but choosing shards in the global cluster has a higher probability of sharing the shard with other apps.
This offers k-anonymity and better connectivity, but comes at a higher bandwidth cost.
Since the introduction of [the Waku Network](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/64/network.md),
it is RECOMMENDED that apps choose a cluster + shard within the defined range for that network.
The name of the pubsub topic corresponding to a given static shard is specified as
`/waku/2/rs/<cluster_id>/<shard_number>`,
@ -88,13 +101,10 @@ an example for the 2nd shard in the global shard cluster:
From an app point of view, a subscription to a content topic `waku2/xxx` on a static shard would look like:
`subscribe("/waku2/xxx", 43)`
for global shard 43.
And for shard 43 of the Status app (which has allocated index 16):
`subscribe("/waku2/xxx", 16, 43)`
for shard 43 of the Status app (which has allocated index 16).
### Discovery
Waku v2 supports the discovery of peers within static shards,
@ -166,9 +176,19 @@ This example node is part of shards `13`, `14`, and `45` in the Status main-net
## Automatic Sharding
Autosharding selects shards automatically and is the default behavior for shard choice.
The other choices being static and named sharding as seen in previous sections.
Shards (pubsub topics) SHOULD be computed from content topics with the procedure below.
Autosharding is an extension of static sharding,
where shards can automatically be selected based on content topic.
This is only possible in cases where there exists a clear definition of a "network" combining multiple shards
with a predefined cluster ID and number of shards in the network (`number_of_shards_in_network`).
An example is [the Waku Network](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/64/network.md).
To use autosharding,
the shards corresponding to the network definition MUST be numbered from `0` and monotonously increase to `number_of_shards_in_network - 1`.
The Waku Network SHOULD be considered the default destination for all app-level protocols.
In cases where the Waku Network is not used but autosharding is desired,
the default number of shards in a network SHOULD be considered `1`.
When using autosharding,
shards (pubsub topics) MUST be computed from content topics with the procedure below.
#### Algorithm
@ -204,9 +224,21 @@ Generation default value is `0`.
- Full length `/0/myapp/1/mytopic/cbor`
- Short length `/myapp/1/mytopic/cbor`
#### Generation
#### Generation and scaling
The generation number monotonously increases and indirectly refers to the total number of shards of the Waku Network.
The generation number monotonously increases and indirectly refers to the total number of shards of a defined network.
In order to scale,
each subsequent generation of a defined network can define a larger `number_of_shards_in_network`,
with the content topics only sharded to the number of shards defined for the corresponding generation of the network.
Generational autosharding MUST be clearly defined for each generation of a network.
For example, consider a specific network defined as the 2 shards `0` and `1` in cluster `32`.
This will automatically be assumed to be generation `0` of this network.
All content topics in the format `/myapp/1/mytopic/cbor` defined for apps on this network will be autosharded to 1 of these 2 shards.
If in future the specifiers of this network want to scale the network to 4 shards (i.e. shards `0` to `3` on cluster `32`),
they MUST define a generation `1` version of the network with `number_of_shards_in_network = 4`.
New content topics for apps on generation `1` of this network MUST be prefixed in the format `/1/myapp/1/mytopic/cbor` to be autosharded into all 4 shards.
Legacy generation `0` content topics will still only be autosharded into the original 2 shards.
<!-- Create a new RFC for each generation spec. -->
@ -282,3 +314,4 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
- [Research log: Waku Discovery](https://vac.dev/wakuv2-apd)
- [WAKU2-RELAY-STATIC-SHARD-ALLOC](../../informational/relay-static-shard-alloc.md)
- [14/WAKU2-MESSAGE](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/14/message.md)
- [64/WAKU2-NETWORK](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/64/network.md)