Background and Motivation # Unstructured P2P networks are more robust and resilient against DoS attacks compared to structured P2P networks). However, they do not scale to large traffic loads. A single libp2p gossipsub mesh, which carries messages associated with a single pubsub topic, can be seen as a separate unstructured P2P network (control messages go beyond these boundaries, but at its core, it is a separate P2P network).">
Background and Motivation # Unstructured P2P networks are more robust and resilient against DoS attacks compared to structured P2P networks). However, they do not scale to large traffic loads. A single libp2p gossipsub mesh, which carries messages associated with a single pubsub topic, can be seen as a separate unstructured P2P network (control messages go beyond these boundaries, but at its core, it is a separate P2P network)." />
However, they do not scale to large traffic loads.
A single <ahref="https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md#gossipsub-the-gossiping-mesh-router">libp2p gossipsub mesh</a>,
which carries messages associated with a single pubsub topic, can be seen as a separate unstructured P2P network
<p><em>Note</em>: Because <em>all</em> shards distribute payload defined in <ahref="spec/14/">14/WAKU2-MESSAGE</a> via <ahref="https://developers.google.com/protocol-buffers/">protocol buffers</a>,
the pubsub topic name does not explicitly add <code>/proto</code> to indicate protocol buffer encoding.
The value is comprised of a two-byte shard cluster index in network byte order concatenated with a 128-byte wide bit vector.
The bit vector indicates which shards of the respective shard cluster the node is part of.
The right-most bit in the bit vector represents shard <code>0</code>, the left-most bit represents shard <code>1023</code>.
The representation in the ENR is inspired by <ahref="https://github.com/ethereum/consensus-specs/blob/dev/specs/altair/validator.md#sync-committee-subnet-stability">Ethereum shard ENRs</a>),
and <ahref="https://github.com/ethereum/consensus-specs/blob/dev/specs/altair/validator.md#sync-committee-subnet-stability">this</a>).</p>
<p>The app is oblivious to the pubsub topic layer.
(Future versions could deprecate the default pubsub topic and remove the necessity for <code>auto=true</code>.)</p>
<p><em>The basic idea behind automatic sharding</em>:
Content topics are mapped using <ahref="https://en.wikipedia.org/wiki/Consistent_hashing">consistent hashing</a>.
Like with DHTs, the hash space is split into parts,
each covered by a Pubsub topic (mesh network) that carries content topics which are mapped into the respective part of the hash space.</p>
<p>There are (at least) two issues that have to be solved: <em>Hot spots</em> and <em>Discovery</em> (see next subsection).</p>
<p>Hot spots occur (similar to DHTs), when a specific mesh network becomes responsible for (several) large multicast groups (content topics).
The opposite problem occurs when a mesh only carries multicast groups with very few participants: this might cause bad connectivity within the mesh.
Our research goal here is finding efficient ways of distribution.
We could get inspired by the DHT literature.
We also have to consider:
If a node is part of many content topics which are all spread over different shards,
the node will potentially be exposed to a lot of network traffic.</p>
<h2id="discovery-1">
Discovery
<aclass="anchor"href="#discovery-1">#</a>
</h2>
<p>For the discovery of automatic shards this document specifies two methods (the second method will be detailed in a future version of this document).</p>
<p>The first method uses the discovery introduced above in the context of static shards.
The index range <code>49152 - 65535</code> is reserved for automatic sharding.
Each index can be seen as a hash bucket.
Consistent hashing maps content topics in one of these buckets.</p>
<p>The second discovery method will be a successor to the first method,
but is planned to preserve the index range allocation.
Instead of adding the data to the ENR, it will treat each array index as a capability,
which can be hierarchical, having each shard in the indexed shard cluster as a sub-capability.
When scaling to a very large number of shards, this will avoid blowing up the ENR size, and allows efficient discovery.
We currently use <ahref="https://rfc.vac.dev/spec/33/">33/WAKU2-DISCV5</a> for discovery,
which is based on Ethereum’s <ahref="https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md">discv5</a>.
While this allows to sample nodes from a distributed set of nodes efficiently and offers good resilience,
it does not allow to efficiently discover nodes with specific capabilities within this node set.
Our <ahref="https://vac.dev/wakuv2-apd">research log post</a> explains this in more detail.
Adding efficient (but still preserving resilience) capability discovery to discv5 is ongoing research.
<ahref="https://github.com/harnen/service-discovery-paper">A paper on this</a> has been completed,
but the <ahref="https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md">Ethereum discv5 specification</a>
has yet to be updated.
When the new capability discovery is available,
this document will be updated with a specification of the second discovery method.
The transition to the second method will be seamless and fully backwards compatible because nodes can still advertise and discover shard memberships in ENRs.</p>
<p>Until automatic sharding is fully specified, (smaller) Apps SHOULD use the default PubSub topic unless there is a good reason not to,
e.g. a requirement to scale to large user numbers (in a rollout phase, the default pubsub topic might still be the better option).</p>
<p>Using a single PubSub topic ensures a connected network, as well as some degree of metadata protection.
See <ahref="/spec/10/#anonymity--unlinkability">section on Anonymity/Unlinkability</a>.</p>
<p>Using another pubsub topic might lead to</p>
<ul>
<li>weaker metadata protection</li>
<li>connectivity problems if there are not enough nodes within the respective pubsub mesh</li>
<li>store nodes might not store messages for the chosen pubsub topic</li>
</ul>
<p>Apps that use named (not the default) or static sharding likely have to setup their own infrastructure nodes which may render the application less robust.</p>
