<metaname="twitter:description"content="A research log. What's the current state of Waku? How many users does it support? What are the bottlenecks? What's next?"/>
<divclass="post__content"><p>Waku is our fork of Whisper where we address the shortcomings of Whisper in an iterative manner. We’ve seen a in <ahref="https://vac.dev/fixing-whisper-with-waku">previous post</a> that Whisper doesn’t scale, and why. In this post we’ll talk about what the current state of Waku is, how many users it can support, and future plans.</p>
<p>We released <ahref="https://rfc.vac.dev/spec/6">Waku spec v0.3</a> this week! You can see the full changelog <ahref="https://rfc.vac.dev/spec/6/#changelog">here</a>.</p>
<p>The main change from 0.2 is making the handshake more flexible. This enables us to communicate topic interest immediately without ambiguity. We also did the following:</p>
<ul>
<li>added recommendation for DNS based discovery</li>
<li>added an upgradability and compatibility policy</li>
<li>cut the spec up into several components</li>
</ul>
<p>We cut the spec up in several components to make Vac as modular as possible. The components right now are:</p>
<p>There are currently two clients that implement Waku v0.3, these are <ahref="https://github.com/status-im/nim-waku">Nimbus (Update: now nim-waku)</a> in Nim and <ahref="https://github.com/status-im/status-go">status-go</a> in Go.</p>
<p>For more details on what each client support and don’t, you can follow the <ahref="https://github.com/vacp2p/pm/issues/7">work in progress checklist</a>.</p>
<p>Work is currently in progress to integrate it into the <ahref="https://github.com/status-im/status-react/pull/9949">Status core app</a>. Waku is expected to be part of their upcoming 1.1 release (see <ahref="https://trello.com/b/DkxQd1ww/status-app-roadmap">Status app roadmap (link deprecated)</a>).</p>
<p>We have a <ahref="https://github.com/status-im/nim-waku/blob/master/waku/v1/node/quicksim.nim">simulation</a> that verifies - or rather, fails to falsify - our <ahref="https://vac.dev/fixing-whisper-with-waku">scalability model</a>. More on the simulation and what it shows below.</p>
<h2id="how-many-users-does-waku-support">How many users does Waku support?</h2>
<p>This is our current understanding of how many users a network running Waku can support. Specifically in the context of the Status chat app, since that’s the most immediate consumer of Waku. It should generalize fairly well to most deployments.</p>
<p><strong>tl;dr (for Status app):</strong></p>
<ul>
<li>beta: 100 DAU</li>
<li>v1: 1k DAU</li>
<li>v1.1 (waku only): 10k DAU (up to x10 with deployment hotfixes)</li>
<li>v1.2 (waku+dns): 100k DAU (can optionally be folded into v1.1)</li>
</ul>
<p><em>Assuming 10 concurrent users = 100 DAU. Estimate uncertainty increases for each order of magnitude until real-world data is observed.</em></p>
<p>As far as we know right now, these are the bottlenecks we have:</p>
<ul>
<li>Immediate bottleneck - Receive bandwidth for end user clients (aka ‘Fixing Whisper with Waku’)</li>
<li>Very likely bottleneck - Nodes and cluster capacity (aka ‘DNS based node discovery’)</li>
<li>Conjecture but not unlikely to appear- Full node traffic (aka ‘the routing / partition problem’)</li>
</ul>
<p>We’ve already seen the first bottleneck being discussed in the initial post. Dean wrote a post on <ahref="https://vac.dev/dns-based-discovery">DNS based discovery</a> which explains how we will address the likely second bottleneck. More on the third one in future posts.</p>
<p>For more details on these bottlenecks, see <ahref="https://discuss.status.im/t/scalability-estimate-how-many-users-can-waku-and-the-status-app-support/1514">Scalability estimate: How many users can Waku and the Status app support?</a>.</p>
<h2id="simulation">Simulation</h2>
<p>The ultimate test is real-world usage. Until then, we have a simulation thanks to Kim De Mey from the Nimbus team!</p>
<p>We have two network topologies, Star and full mesh. Both networks have 6 full nodes, one traditional light node with bloom filter, and one Waku light node.</p>
<p>One of the full nodes sends 1 envelope over 1 of the 100 topics that the two light nodes subscribe to. After that, it sends 10000 envelopes over random topics.</p>
<p>For light node, bloom filter is set to almost 10% false positive (bloom filter: n=100, k=3, m=512). It shows the number of valid and invalid envelopes received for the different nodes.</p>
<p><strong>Star network:</strong></p>
<table>
<thead>
<tr>
<th>Description</th>
<th>Peers</th>
<th>Valid</th>
<th>Invalid</th>
</tr>
</thead>
<tbody>
<tr>
<td>Master node</td>
<td>7</td>
<td>10001</td>
<td>0</td>
</tr>
<tr>
<td>Full node 1</td>
<td>3</td>
<td>10001</td>
<td>0</td>
</tr>
<tr>
<td>Full node 2</td>
<td>1</td>
<td>10001</td>
<td>0</td>
</tr>
<tr>
<td>Full node 3</td>
<td>1</td>
<td>10001</td>
<td>0</td>
</tr>
<tr>
<td>Full node 4</td>
<td>1</td>
<td>10001</td>
<td>0</td>
</tr>
<tr>
<td>Full node 5</td>
<td>1</td>
<td>10001</td>
<td>0</td>
</tr>
<tr>
<td>Light node</td>
<td>2</td>
<td>815</td>
<td>0</td>
</tr>
<tr>
<td>Waku light node</td>
<td>2</td>
<td>1</td>
<td>0</td>
</tr>
</tbody>
</table>
<p><strong>Full mesh:</strong></p>
<table>
<thead>
<tr>
<th>Description</th>
<th>Peers</th>
<th>Valid</th>
<th>Invalid</th>
</tr>
</thead>
<tbody>
<tr>
<td>Full node 0</td>
<td>7</td>
<td>10001</td>
<td>20676</td>
</tr>
<tr>
<td>Full node 1</td>
<td>7</td>
<td>10001</td>
<td>9554</td>
</tr>
<tr>
<td>Full node 2</td>
<td>5</td>
<td>10001</td>
<td>23304</td>
</tr>
<tr>
<td>Full node 3</td>
<td>5</td>
<td>10001</td>
<td>11983</td>
</tr>
<tr>
<td>Full node 4</td>
<td>5</td>
<td>10001</td>
<td>24425</td>
</tr>
<tr>
<td>Full node 5</td>
<td>5</td>
<td>10001</td>
<td>23472</td>
</tr>
<tr>
<td>Light node</td>
<td>2</td>
<td>803</td>
<td>803</td>
</tr>
<tr>
<td>Waku light node</td>
<td>2</td>
<td>1</td>
<td>1</td>
</tr>
</tbody>
</table>
<p>Things to note:</p>
<ul>
<li>Whisper light node with ~10% false positive gets ~10% of total traffic</li>
<li>Waku light node gets ~1000x less envelopes than Whisper light node</li>
<li>Full mesh results in a lot more duplicate messages, expect for Waku light node</li>
<p>Run the simulation yourself <ahref="https://github.com/status-im/nim-waku/blob/master/waku/v1/node/quicksim.nim">here</a>. The parameters are configurable, and it is integrated with Prometheus and Grafana.</p>
<p>For the third bottleneck, a likely candidate for fixing this is Kademlia routing. This is similar to what is done in <ahref="https://www.ethswarm.org/">Swarm’s</a> PSS. We are in the early stages of experimenting with this over libp2p in <ahref="https://github.com/status-im/nim-libp2p">nim-libp2p</a>. More on this in a future post!</p>
<p><em>Image from “caged sky” by mh.xbhd.org is licensed under CC BY 2.0 (https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e)</em></p>
