roadmap/content/waku/updates/2023-11-06.md

145 lines
8.7 KiB
Markdown
Raw Normal View History

2023-11-02 09:18:53 +00:00
---
2023-11-02 09:20:36 +00:00
title: 2023-11-06 Waku weekly
2023-11-02 09:18:53 +00:00
tags:
- waku-updates
2023-11-02 09:20:36 +00:00
date: 2023-11-06
2023-11-07 15:00:20 +00:00
draft: false
2023-11-02 09:18:53 +00:00
---
2023-11-06 01:33:45 +00:00
# 2023-11-06 Waku weekly
## [Waku Network Can Support 10K Users](https://github.com/waku-org/pm/issues/12)
- _achieved:_
2023-11-07 14:59:51 +00:00
- further PostgreSQL optimisations nearing conclusion
- implemented bridge to allow Status Community to move to static sharding with backwards compatibility towards default pubsub topic
- solution for shared bootstrap nodes being filtered out in discv5 as more static shards are activated
- ensured no unknown blockers from Waku's side to start dogfooding in conversation with Status Communities
2023-11-06 01:33:45 +00:00
- _next:_
2023-11-07 14:59:51 +00:00
- continue integration of static sharding in status-go.
- deploy bridge for backwards compatibility
- dogfooding of Status Desktop with Status staging fleet. Will aim to create a small internal Waku community
2023-11-06 01:33:45 +00:00
- _risks:_
2023-11-07 14:59:51 +00:00
- Dependency on Vac/DST to conclude ~1k nodes simulations.
- Implementation of static sharding in Status Communities and design decisions mostly driven by go-waku developer, with minimal input from Status dev ([1](https://github.com/status-im/status-go/pull/4161), [2](https://github.com/status-im/status-go/pull/4094), [3](https://github.com/status-im/status-go/pull/4093)). See [status-go#4057](https://github.com/status-im/status-go/issues/4057) for remaining work. Mitigation by on-boarding Chat SDK lead on 6 Nov to drive effort.
- lack of confidence in simulation results: results so far exhibits various artifacts and anomalies seemingly related to tooling limitations. It is therefore difficult to draw certain conclusions re Waku scalability.
- lack of clarity in terms of Status fleet ownership, monitoring and maintenance, which is an integral part of the solution.
2023-11-06 01:33:45 +00:00
### [Targeted dogfooding for Status Communities](https://github.com/waku-org/pm/issues/97)
- _achieved_: fix in status-go to use correct unprotected pubsub topic for community point of contacts, added pubsub topic bridge feature to go-waku, stop filtering out bootnodes on discovery, minimize noise on logs when selecting peers and not having peers available.
- _next_: deploy bridge between default pubsub topic and shard 32
## [Waku Network can Support 1 Million Users](https://github.com/waku-org/pm/issues/83) - 2023-11-30
- _achieved_:
2023-11-07 14:59:51 +00:00
- See 10k milestone update for PostgreSQL status.
- First version of the 10k-tool by DST is ready and is being tested with simulation running a small nim-libp2p/gossipsub binary.
2023-11-06 01:33:45 +00:00
- _risks_:
2023-11-07 14:59:51 +00:00
- Dependency on Vac/DST to run 10k nodes simulations. Tracked under
[`vac:dst:eng-10ktool`](https://roadmap.logos.co/tags/vac-updates).
- Wakutorsis tool is being dropped, meaning new tooling needs to be developed for 10k nodes simulations. It is currently uncertain whether such tool can be developed.
- Large scale simulations done by Vac/DST only covered nwaku relay. go-waku, status-go simulations are not planned short term (theoretical review of Status Communities messages is), nor are simulations including request-response protocols such as store and filter.
- lack of real world feedback/dogfooding: the complete static sharding solution involves significant changes to the Waku protocol and tech stack. Although each element is unit tested, dogfooding may hit corner cases in the integrated solution that cannot be foreseen/recreated in lab conditions.
2023-11-06 01:33:45 +00:00
### [PostgreSQL in service node: Further optimisations](https://github.com/waku-org/pm/issues/84)
**[nwaku] [PostgreSQL](https://github.com/waku-org/nwaku/issues/1888)**
- _achieved_: Optimize _select_/_Store_ queries by adding prepared statements. [PR](https://github.com/waku-org/nwaku/pull/2182)
- _next_: Wrap up the Postgres optimizations. Summarize the performance comparison in a report.
**[nwaku] [PostgreSQL](https://github.com/waku-org/nwaku/issues/1888)**
- _achieved_:
2023-11-07 14:59:51 +00:00
- Time processing enhancement when performing _SELECT_ operations. There was an overhead caused by looping too many times over the returned rows, in order to convert the row types. By applying a "rowCallback" approach we can reduce by 30ms the time spent on the query under analysis.
2023-11-06 01:33:45 +00:00
- _next_:
2023-11-07 14:59:51 +00:00
- The queries used in the comparison analysis still perform much better in _SQLite_ (< ~5ms) than in _Postgres_ (< ~15ms.) Therefore we need to push the investigation further to enhance that.
( Edited: notice that the timings indicated above are for tests using consecutive queries. If the queries are performed concurrently, then the timings are worse. I will elaborate more in a report shortly .)
2023-11-06 01:33:45 +00:00
## [Waku Network Gen 0](https://github.com/waku-org/pm/issues/50) - 2023-12-01
- _achieved_:
2023-11-07 14:59:51 +00:00
- Starting internal dogfooding of Waku Network: We have launched the proto-network and it is already possible to run a node in said network.
2023-11-06 01:33:45 +00:00
- _risks_:
2023-11-07 14:59:51 +00:00
- Usage of RLN in js-waku and dependency on a (centralized?) Web3Provider remains unclear as one needs to know the merkle tree state (on chain) to generate proofs.
- We are progressively moving a nwaku engineer to a solution engineer role we need to backfill the role.
- js-waku team is juggling between dev ex and gen 0 with only 2 engineers (3rd one just joined) so delivery in this client is likely to lag behind other clients.
- Uncertainty as to how RLN membership mechanism would hinder application adoption, if memberships need to be distributed or obtained by registration, if staking is necessary to prevent abuse, etc.
2023-11-06 01:33:45 +00:00
### [3.2: Basic DoS protection in production](https://github.com/waku-org/pm/issues/70)
**[nwaku] [feat: add rln to waku simulator instance](https://github.com/waku-org/nwaku/issues/2143)**
- _achieved_: integrated RLN, added Grafana dashboards, tested, merged and deployed
### [1.4: Sharded peer management and discovery](https://github.com/waku-org/pm/issues/67)
**[go-waku] [feat: Service peer selection on specific shards](https://github.com/waku-org/go-waku/issues/680)**
- _achieved_: on-demand peer discovery for service peer selection and new relay shard subscriptions
### [2.3: Basic distributed Store services](https://github.com/waku-org/pm/issues/64)
**[nwaku] [feat: add new message_hash column to the archive protocol](https://github.com/waku-org/nwaku/issues/2112)**
- _achieved_: PR to support SQLite code to support `messageHash` attribute without interrupting the existing cursor-related functionality, `id` field stays for now. Skelton for sync in progress.
- _next_:
2023-11-07 14:59:51 +00:00
- finalize the SQLite `messageHash` attribute and add a research page about it.
- start a research page about the sync mechanism for nWaku, doing request/reply a PoC on the same.
2023-11-06 01:33:45 +00:00
### [2.1: Production testing of existing protocols](https://github.com/waku-org/pm/issues/49)
**[nwaku] [PostgreSQL](https://github.com/waku-org/nwaku/issues/1888)**
- _achieved_: Optimize _select_/_Store_ queries by adding prepared statements. [PR](https://github.com/waku-org/nwaku/pull/2182)
- _next_: Wrap up the Postgres optimizations. Summarize the performance comparison in a report.
**[nwaku] [PostgreSQL](https://github.com/waku-org/nwaku/issues/1888)**
- _achieved_:
2023-11-07 14:59:51 +00:00
- Time processing enhancement when performing _SELECT_ operations. There was an overhead caused by looping too many times over the returned rows, in order to convert the row types. By applying a "rowCallback" approach we can reduce by 30ms the time spent on the query under analysis.
2023-11-06 01:33:45 +00:00
- _next_:
2023-11-07 14:59:51 +00:00
- The queries used in the comparison analysis still perform much better in _SQLite_ (< ~5ms) than in _Postgres_ (< ~15ms.) Therefore we need to push the investigation further to enhance that.
( Edited: notice that the timings indicated above are for tests using consecutive queries. If the queries are performed concurrently, then the timings are worse. I will elaborate more in a report shortly .)
2023-11-06 01:33:45 +00:00
## [Quality Assurance processes are in place](https://github.com/waku-org/pm/issues/73) - 2024-03-31
## [Support Many Platforms](https://github.com/waku-org/pm/issues/42) - 2024-04-30
### [Presentation Readiness ](https://github.com/waku-org/pm/issues/95)
**[js-waku] [feat: better support for development](https://github.com/waku-org/js-waku/issues/1665)**
- _achieved_: by going through basic flow identified errors that can or cannot be ignored;
- _next_: working on improving log errors, aiming to complete in couple of days;
2023-11-06 08:26:30 +00:00
# Ecosystem Development
- _achieved_:
2023-11-07 14:59:51 +00:00
- Multiple leads from ETHLisbon
- Sync call with The Graph
2023-11-06 08:26:30 +00:00
- _next_:
2023-11-07 14:59:51 +00:00
- Review RLN docs readiness for launch at Devconnect
- Web3privacy event preparation
- _achieved_:
- Multiple leads from ETHLisbon.
- Sync call with The Graph.
- js-waku prioritising.
- Created Hackathon project with Waku at ETHLisbon.
- Awesome-waku updates.
- _next_:
- Review RLN docs readiness for launch at DevConnect.
- Web3privacy event preparation.
- Waku network soft-launch organisation.
- Creating tutorial videos.
- ETHStaker gathering for gaining some awareness about node operator onboarding.