waku-org
/
specs
mirror of https://github.com/waku-org/specs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update relay-sharding.md

This commit is contained in:
Jimmy Debe 2024-07-09 06:43:06 -04:00 committed by GitHub
parent d4f5b5bf57
commit e3261c99bb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 11 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
standards/core/relay-sharding.md
View File

@ -11,7 +11,7 @@ contributors:
## Abstract
This document describes ways of sharding the [Waku relay](https://rfc.vac.dev/waku/standards/core/11/relay) topic,
This document describes ways of sharding the [Waku relay](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/11/relay.md) topic,
allowing Waku networks to scale in the number of content topics.
> _Note_: Scaling in the size of a single content topic is out of scope for this document.
@ -25,7 +25,7 @@ However, they do not scale to large traffic loads.
A single [libp2p gossipsub mesh](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md#gossipsub-the-gossiping-mesh-router),
which carries messages associated with a single pubsub topic, can be seen as a separate unstructured P2P network
(control messages go beyond these boundaries, but at its core, it is a separate P2P network).
With this, the number of [Waku relay](https://rfc.vac.dev/waku/standards/core/11/relay) content topics that can be carried over a pubsub topic is limited.
With this, the number of [Waku relay](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/11/relay.md) content topics that can be carried over a pubsub topic is limited.
This prevents app protocols that aim to span many multicast groups (realized by content topics) from scaling.
This document specifies three pubsub topic sharding methods (with varying degrees of automation),
@ -35,7 +35,7 @@ This document also covers discovery of topic shards.
## Named Sharding
_Named sharding_ offers apps to freely choose pubsub topic names.
It is RECOMMENDED for App protocols to follow the naming structure detailed in [23/WAKU2-TOPICS](https://rfc.vac.dev/waku/informational/23/topics).
It is RECOMMENDED for App protocols to follow the naming structure detailed in [23/WAKU2-TOPICS](https://github.com/vacp2p/rfc-index/blob/main/waku/informational/23/topics.md).
With named sharding, managing discovery falls into the responsibility of apps.
From an app protocol point of view, a subscription to a content topic `waku2/xxx` on a shard named /mesh/v1.1.1/xxx would look like:
@ -82,7 +82,7 @@ an example for the 2nd shard in the global shard cluster:
`/waku/2/rs/0/2`.
> _Note_: Because _all_ shards distribute payload defined in [14/WAKU2-MESSAGE](https://rfc.vac.dev/waku/standards/core/14/message) via [protocol buffers](https://developers.google.com/protocol-buffers/),
> _Note_: Because _all_ shards distribute payload defined in [14/WAKU2-MESSAGE](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/14/message.md) via [protocol buffers](https://developers.google.com/protocol-buffers/),
> the pubsub topic name does not explicitly add `/proto` to indicate protocol buffer encoding.
> We use `rs` to indicate these are _relay shard_ clusters; further shard types might follow in the future.
@ -102,7 +102,7 @@ so app protocols do not have to implement their own discovery method.
Nodes add information about their shard participation in their [WAKU2-ENR](./enr.md).
Having a static shard participation indication as part of the ENR allows nodes
to discover peers that are part of shards via [33/WAKU2-DISCV5](https://rfc.vac.dev/waku/standards/core/33/discv5) as well as via DNS.
to discover peers that are part of shards via [33/WAKU2-DISCV5](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/33/discv5.md) as well as via DNS.
> _Note:_ In the current version of this document,
> sharding information is directly added to the ENR.
@ -191,7 +191,7 @@ The shard to use is the modulo of the hash by the number of shards in the networ
### Content Topics Format for Autosharding
Content topics MUST follow the format in [23/WAKU2-TOPICS](https://rfc.vac.dev/waku/informational/23/topics/#content-topic-format).
Content topics MUST follow the format in [23/WAKU2-TOPICS](https://github.com/vacp2p/rfc-index/blob/main/waku/informational/23/topics.md/#content-topic-format).
In addition, a generation prefix MAY be added to content topics.
When omitted default values are used.
Generation default value is `0`.
@ -242,7 +242,7 @@ but is planned to preserve the index range allocation.
Instead of adding the data to the ENR, it will treat each array index as a capability,
which can be hierarchical, having each shard in the indexed shard cluster as a sub-capability.
When scaling to a very large number of shards, this will avoid blowing up the ENR size, and allows efficient discovery.
We currently use [33/WAKU2-DISCV5](https://rfc.vac.dev/waku/standards/core/33/discv5) for discovery,
We currently use [33/WAKU2-DISCV5](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/33/discv5.md) for discovery,
which is based on Ethereum's [discv5](https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md).
While this allows to sample nodes from a distributed set of nodes efficiently and offers good resilience,
it does not allow to efficiently discover nodes with specific capabilities within this node set.
@ -272,12 +272,13 @@ Copyright and related rights waived via [CC0](https://creativecommons.org/public
## References
- [11/WAKU2-RELAY](https://rfc.vac.dev/waku/standards/core/11/relay)
- [11/WAKU2-RELAY](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/11/relay.md)
- [Unstructured P2P network](https://en.wikipedia.org/wiki/Peer-to-peer#Unstructured_networks)
- [33/WAKU2-DISCV5](https://rfc.vac.dev/waku/standards/core/33/discv5)
- [33/WAKU2-DISCV5](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/33/discv5.md)
- [WAKU2-ENR](./enr.md)
- [23/WAKU2-TOPICS](https://rfc.vac.dev/waku/informational/23/topics)
- [23/WAKU2-TOPICS](https://github.com/vacp2p/rfc-index/blob/main/waku/informational/23/topics.md)
- [Ethereum ENR sharding bit vector](https://github.com/ethereum/consensus-specs/blob/dev/specs/altair/p2p-interface.md#metadata)
- [Ethereum discv5 specification](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md)
- [Research log: Waku Discovery](https://vac.dev/wakuv2-apd)
- [WAKU2-RELAY-STATIC-SHARD-ALLOC](../../informational/relay-static-shard-alloc.md)
- [14/WAKU2-MESSAGE](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/14/message.md)