2023-11-20 Waku weekly

Waku Network Can Support 10K Users

achieved:
- closed last PostgreSQL issue for Store scalability
- confirmed no unknown blockers from Waku's side to continue dogfooding in conversation with Status Communities
- started team-internal dogfooding of a test community using static sharding
- started fleet ownership handover process: published guidelines/list of responsibilities - https://www.notion.so/Fleet-Ownership-7532aad8896d46599abac3c274189741
next:
- continue dogfooding of Status Desktop with Status staging fleet with test community
- training session to conclude fleet ownership handover: https://www.notion.so/Fleet-Ownership-7532aad8896d46599abac3c274189741
risks:
- Dependency on Vac/DST to conclude ~1k nodes simulations.
- Implementation of static sharding in Status Communities and design decisions mostly driven by go-waku developer, with minimal input from Status dev (1, 2, 3). See status-go#4057 for remaining work. Mitigation by on-boarding Chat SDK lead on 6 Nov to drive effort.
- lack of confidence in simulation results: results so far exhibits various artifacts and anomalies seemingly related to tooling limitations. It is therefore difficult to draw certain conclusions re Waku scalability.
- lack of clarity in terms of Status fleet ownership, monitoring and maintenance, which is an integral part of the solution.

achieved: logout / login freeze, fix request on correct pubsub topic, and add missing shard information on community invite
next: dogfooding

achieved:
- Closed last Postgresql issue for basic Store scalability. See 10k milestone.
- Assisted DST in setting up initial tests with the ~1K tool. Currently still fine-tuning parameters, ensuring results are consistent, etc. for smaller configurations.
risks:
- Dependency on Vac/DST to run 10k nodes simulations. Tracked under vac:dst:eng-10ktool.
- Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
- Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
- lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.

achieved:
- Internal dogfooding of proto-network continues.
- Significant progress of autosharding in js-waku: autosharding function implemented, work to integrate in protocols started.
risks:
- Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
- We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
- js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
- Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc.

achieved: derive pubsub topic from content topic in encoders/decoders when autosharding is specified
next: node config should specify static sharding or autosharding. implement autosharded topics in all req-resp protocols

achieved: modified ConnectionManager to track connection state. exposed through read function and new event
next: resolve any remaining feedback

achieved: Better support for js-waku using RestApi: Allow text/plain contentType
next: Bandwidth management preparation