<html><head><title>2023-10-30 Waku weekly</title><metacharSet="utf-8"/><metaname="viewport"content="width=device-width, initial-scale=1.0"/><metaproperty="og:title"content="2023-10-30 Waku weekly"/><metaproperty="og:description"content="2023-10-30 Waku weekly § Waku Network Can Support 10K Users § Integration of static sharding in go-waku is continuing (see updates below). Testing of PostgreSQL enabled some performance improvement in the implementation that are being implemented."/><metaproperty="og:image"content="https://roadmap.logos.co/static/og-image.png"/><metaproperty="og:width"content="1200"/><metaproperty="og:height"content="675"/><linkrel="icon"href="../../static/icon.png"/><metaname="description"content="2023-10-30 Waku weekly § Waku Network Can Support 10K Users § Integration of static sharding in go-waku is continuing (see updates below). Testing of PostgreSQL enabled some performance improvement in the implementation that are being implemented."/><metaname="generator"content="Quartz"/><linkrel="preconnect"href="https://fonts.googleapis.com"/><linkrel="preconnect"href="https://fonts.gstatic.com"/><linkhref="../../index.css"rel="stylesheet"type="text/css"spa-preserve/><linkhref="https://cdn.jsdelivr.net/npm/katex@0.16.0/dist/katex.min.css"rel="stylesheet"type="text/css"spa-preserve/><linkhref="https://fonts.googleapis.com/css2?family=IBM Plex Mono&family=sans-serif:wght@400;700&family=sans-serif:ital,wght@0,400;0,600;1,400;1,600&display=swap"rel="stylesheet"type="text/css"spa-preserve/><scriptsrc="../../prescript.js"type="application/javascript"spa-preserve></script><scripttype="application/javascript"spa-preserve>constfetchData=fetch(`../../static/contentIndex.json`).then(data=>data.json())</script></head><bodydata-slug="waku/updates/2023-10-30"><divid="quartz-root"class="page"><divid="quartz-body"><divclass="left sidebar"><h1class="page-title "><ahref="../..">Logos Collective Project Roadmaps</a></h1><divclass="spacer mobile-only"></div><divclass="search "><divid="search-icon"><p>Search</p><div></div><svgtabIndex="0"aria-labelledby="title desc"role="img"xmlns="http://www.w3.org/2000/svg"viewBox="0 0 19.9 19.7"><titleid="title">Search</title><descid="desc">Search</desc><gclass="search-path"fill="none"><pathstroke-linecap="square"d="M18.5 18.3l-5.4-5.4"></path><circlecx="8"cy="8"r="7"></circle></g></svg></div><divid="search-container"><divid="search-space"><inputautocomplete="off"id="search-bar"name="search"type="text"aria-label="Search for something"placeholder="Search for something"/><divid="results-container"></div></div></div></div><divclass="darkmode "><inputclass="toggle"id="darkmode-toggle"type="checkbox"tabIndex="-1"/><labelid="toggle-label-light"for="darkmode-toggle"tabIndex="-1"><svgxmlns="http://www.w3.org/2000/svg"xmlnsXlink="http://www.w3.org/1999/xlink"version="1.1"id="dayIcon"x="0px"y="0px"viewBox="0 0 35 35"style="enable-background:new 0 0 35 35;"xmlSpace="preserve"><title>Light mode</title><pathd="M6,17.5C6,16.672,5.328,16,4.5,16h-3C0.672,16,0,16.672,0,17.5S0.672,19,1.5,19h3C5.328,19,6,18.328,6,17.5zM7.5,26c-0.414,0-0.789,0.168-1.061,0.439l-2,2C4.168,28.711,4,29.086,4,29.5C4,30.328,4.671,31,5.5,31c0.414,0,0.789-0.168,1.06-0.44l2-2C8.832,28.289,9,27.914,9,27.5C9,26.672,8.329,26,7.5,26zM17.5,6C18.329,6,19,5.328,19,4.5v-3C19,0.672,18.329,0,17.5,0S16,0.672,16,1.5v3C16,5.328,16.671,6,17.5,6zM27.5,9c0.414,0,0.789-0.168,1.06-0.439l2-2C30.832,6.289,31,5.914,31,5.5C31,4.672,30.329,4,29.5,4c-0.414,0-0.789,0.168-1.061,0.44l-2,2C26.168,6.711,26,7.086,26,7.5C26,8.328,26.671,9,27.5,9zM6.439,8.561C6.711,8.832,7.086,9,7.5,9C8.328,9,9,8.328,9,7.5c0-0.414-0.168-0.789-0.439-1.061l-2-2C6.289,4.168,5.914,4,5.5,4C4.672,4,4,4.672,4,5.5c0,0.414,0.168,0.789,0.439,1.06L6.439,8.561zM33.5,16h-3c-0.828,0-1.5,0.672-1.5,1.5s0.672,1.5,1.5,1.5h3c0.828,0,1.5-0.672,1.5-1.5S34.328,16,33.5,16zM28.561,26.439C28.289,26.168,27.914,26,27.5,26c-0.828,0-1.5,0.672-1.5,1.5c0,0.414,0.168,0.
<h2id="waku-network-can-support-10k-users"><ahref="https://github.com/waku-org/pm/issues/12"class="external">Waku Network Can Support 10K Users</a><aaria-hidden="true"tabindex="-1"href="#waku-network-can-support-10k-users"class="internal"> §</a></h2>
<li>Integration of static sharding in go-waku is continuing (see updates below).</li>
<li>Testing of PostgreSQL enabled some performance improvement in the implementation that are being implemented.</li>
<li>Internal instructions have been distributed to dogfood static sharding with the Waku team (<ahref="https://discord.com/channels/1110799176264056863/1166125031793901589/1166125169480306708"class="external">Waku Discord private channel</a>).</li>
<li>Dependency on Vac/DST to conclude ~1k nodes simulations.</li>
<li>Implementation of static sharding in Status Communities and design decisions mostly driven by go-waku developer, with minimal input from Status dev (<ahref="https://github.com/status-im/status-go/pull/4161"class="external">1</a>, <ahref="https://github.com/status-im/status-go/pull/4094"class="external">2</a>, <ahref="https://github.com/status-im/status-go/pull/4093"class="external">3</a>). See <ahref="https://github.com/status-im/status-go/issues/4057"class="external">status-go#4057</a> for remaining work. Mitigation by on-boarding Chat SDK lead on 6 Nov to drive effort.</li>
</ul>
</li>
</ul>
<h3id="targeted-dogfooding-for-status-communities"><ahref="https://github.com/waku-org/pm/issues/97"class="external">Targeted dogfooding for Status Communities</a><aaria-hidden="true"tabindex="-1"href="#targeted-dogfooding-for-status-communities"class="internal"> §</a></h3>
<ul>
<li><em>achieved</em>: unsuccesfully tried to avoid introducing a <ahref="https://github.com/status-im/status-go/pull/4161"class="external">breaking change</a> in status-go. We need to decide whether to go ahead and merge that PR</li>
<li>blocker: discv5 filters out outdated ENR entries from DNS Discovery URL in shard fleet - <ahref="https://github.com/waku-org/nwaku/issues/2162"class="external">https://github.com/waku-org/nwaku/issues/2162</a></li>
</ul>
<h2id="waku-network-can-support-1-million-users---2023-11-30"><ahref="https://github.com/waku-org/pm/issues/83"class="external">Waku Network can Support 1 Million Users</a> - 2023-11-30<aaria-hidden="true"tabindex="-1"href="#waku-network-can-support-1-million-users---2023-11-30"class="internal"> §</a></h2>
<ul>
<li><em>achieved</em>:
<ul>
<li>See 10k milestone update for PostgreSQL status.</li>
<li>First version of the 10k-tool by DST is ready and is being tested with simulation running a small nim-libp2p/gossipsub binary.</li>
</ul>
</li>
<li><em>risks</em>:
<ul>
<li>Dependency on Vac/DST to run 10k nodes simulations. Tracked under
<li>Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.</li>
<li>Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.</li>
</ul>
</li>
</ul>
<h3id="postgresql-in-service-node-further-optimisations"><ahref="https://github.com/waku-org/pm/issues/84"class="external">PostgreSQL in service node: Further optimisations</a><aaria-hidden="true"tabindex="-1"href="#postgresql-in-service-node-further-optimisations"class="internal"> §</a></h3>
<li>Time processing enhancement when performing <em>SELECT</em> operations. There was an overhead caused by looping too many times over the returned rows, in order to convert the row types. By applying a “rowCallback” approach we can reduce by 30ms the time spent on the query under analysis.</li>
<li>The queries used in the comparison analysis still perform much better in <em>SQLite</em> (< ~5ms) than in <em>Postgres</em> (< ~15ms.) Therefore we need to push the investigation further to enhance that.</li>
</ul>
</li>
</ul>
<h2id="waku-network-gen-0---2023-12-01"><ahref="https://github.com/waku-org/pm/issues/50"class="external">Waku Network Gen 0</a> - 2023-12-01<aaria-hidden="true"tabindex="-1"href="#waku-network-gen-0---2023-12-01"class="internal"> §</a></h2>
<ul>
<li><em>achieved</em>:
<ul>
<li>Further simulation done, with a continued focus on message propagation time and possible improvements.</li>
<li>Progress across all client son sharded peer management discovery</li>
<li>First PRs merged towards basic distributed store services</li>
</ul>
</li>
<li><em>risks</em>:
<ul>
<li>Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.</li>
<li>We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.</li>
<li>js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.</li>
</ul>
</li>
</ul>
<h3id="32-basic-dos-protection-in-production"><ahref="https://github.com/waku-org/pm/issues/70"class="external">3.2: Basic DoS protection in production</a><aaria-hidden="true"tabindex="-1"href="#32-basic-dos-protection-in-production"class="internal"> §</a></h3>
<p><strong>[nwaku] <ahref="https://github.com/waku-org/nwaku/issues/2143"class="external">feat: add rln to waku simulator instance</a></strong></p>
<ul>
<li><em>achieved</em>: learnt about waku-simulator’s inner workings and got the background required to integrate RLN to it. Added service that generates traffic to the nodes via their REST APIs. Investigated and tested different ways of approaching the RLN integration.</li>
<li><em>next</em>: get RLN to work and add Grafana dashboards with RLN data</li>
<p><strong>[research] <ahref="https://github.com/waku-org/research/issues/44"class="external">Tuning GossipSub’s D parameter in Waku</a></strong></p>
<ul>
<li><em>achieved</em>: nwaku simulations showing the impact in message propagation delay when reducing gossipsub’s D value. Main goal is to reduce bandwidth consumption in exchange of worsen propagation delay.</li>
<li><em>next</em>: asses if we want to move forward changing D.</li>
</ul>
<p><strong>[research] <ahref="https://github.com/waku-org/research/issues/42"class="external">Message propagation times with waku-rln</a></strong></p>
<ul>
<li><em>achieved</em>: Final simulation results with 1000 nwaku nodes with rln enabled, with the goal of measuring message propagation delays under different conditions (amount of nodes and message size).</li>
<li><em>next</em>: NA</li>
</ul>
<h3id="14-sharded-peer-management-and-discovery"><ahref="https://github.com/waku-org/pm/issues/67"class="external">1.4: Sharded peer management and discovery</a><aaria-hidden="true"tabindex="-1"href="#14-sharded-peer-management-and-discovery"class="internal"> §</a></h3>
<p><strong>[nwaku] <ahref="https://github.com/waku-org/nwaku/issues/1941"class="external">feat: Service peer selection on specific shards</a></strong></p>
<p><strong>[nwaku] <ahref="https://github.com/waku-org/nwaku/issues/1940"class="external">feat: Peer management with shard as a dimension</a></strong></p>
<li><em>next</em>: finalize sharded peer store pruning & run simulations</li>
</ul>
<p><strong>[go-waku] <ahref="https://github.com/waku-org/go-waku/issues/842"class="external">feat: Deprecate Named Sharding and Update Lightpush Client API</a></strong></p>
<ul>
<li><em>achieved</em>: Create PR for review for removing of Named Pubsubtopic.</li>
</ul>
<p><strong>[go-waku] <ahref="https://github.com/waku-org/go-waku/issues/680"class="external">feat: Service peer selection on specific shards</a></strong></p>
<ul>
<li><em>achieved</em>: draft PR #834 opened for on-demand peer discovery</li>
<li><em>next</em>: use on-demand peer discovery for service and relay peer selection</li>
</ul>
<h3id="23-basic-distributed-store-services"><ahref="https://github.com/waku-org/pm/issues/64"class="external">2.3: Basic distributed Store services</a><aaria-hidden="true"tabindex="-1"href="#23-basic-distributed-store-services"class="internal"> §</a></h3>
<p><strong>[nwaku] <ahref="https://github.com/waku-org/nwaku/issues/2112"class="external">feat: add new message_hash column to the archive protocol</a></strong></p>
<ul>
<li><em>achieved</em>: On SQLite’s schema transition (i.e. this PR) to <code>messageHash</code> feature complete PR posted (awaiting reviews), Gained insight into the connection and interplay between the store and archive components, and how they may be leveraged into making a sync protocol. Small stuff - bug fix on the jsWaku which was this PR dependent (that too was time-consuming since my first time interacting with JS code of waku), PR on vacuum on time-based retention policy, thought through the nitty gritty details of node based roles and incentives.</li>
<h3id="21-production-testing-of-existing-protocols"><ahref="https://github.com/waku-org/pm/issues/49"class="external">2.1: Production testing of existing protocols</a><aaria-hidden="true"tabindex="-1"href="#21-production-testing-of-existing-protocols"class="internal"> §</a></h3>
<li>Time processing enhancement when performing <em>SELECT</em> operations. There was an overhead caused by looping too many times over the returned rows, in order to convert the row types. By applying a “rowCallback” approach we can reduce by 30ms the time spent on the query under analysis.</li>
<li>The queries used in the comparison analysis still perform much better in <em>SQLite</em> (< ~5ms) than in <em>Postgres</em> (< ~15ms.) Therefore we need to push the investigation further to enhance that.</li>
<li><em>achieved</em>: fix issues found during testing</li>
<li><em>next</em>: publish new version</li>
</ul>
<h2id="quality-assurance-processes-are-in-place---2024-03-31"><ahref="https://github.com/waku-org/pm/issues/73"class="external">Quality Assurance processes are in place</a> - 2024-03-31<aaria-hidden="true"tabindex="-1"href="#quality-assurance-processes-are-in-place---2024-03-31"class="internal"> §</a></h2>
<h2id="support-many-platforms---2024-04-30"><ahref="https://github.com/waku-org/pm/issues/42"class="external">Support Many Platforms</a> - 2024-04-30<aaria-hidden="true"tabindex="-1"href="#support-many-platforms---2024-04-30"class="internal"> §</a></h2>
<h3id="ship-rln-as-part-of-non-native-sdks"><ahref="https://github.com/waku-org/pm/issues/88"class="external">Ship RLN as part of non-native SDKs</a><aaria-hidden="true"tabindex="-1"href="#ship-rln-as-part-of-non-native-sdks"class="internal"> §</a></h3>
<p><strong>[go-waku] <ahref="https://github.com/waku-org/go-waku/issues/788"class="external">refactor: add user_data to c-bindings</a></strong></p>
<ul>
<li><em>achieved</em>: fixed issues found during tests with waku-rust-bindings</li>
</ul>
<h3id="rest-api-service-node"><ahref="https://github.com/waku-org/pm/issues/82"class="external">REST API service node</a><aaria-hidden="true"tabindex="-1"href="#rest-api-service-node"class="internal"> §</a></h3>
