vacp2p
/
vac.dev-experimental-old
mirror of https://github.com/vacp2p/vac.dev-experimental-old.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Divide posts into categories

This commit is contained in:
Maria Rushkova 2021-08-06 12:17:19 +02:00 committed by Jakub Sokołowski
parent be5f875a13
commit 616ef9fa68
No known key found for this signature in database
GPG Key ID: 4EF064D0E6D63020
20 changed files with 1054 additions and 296 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
_layouts/research-log.html
View File

@ -61,7 +61,8 @@ layout: default
<div class="info-block w-full sm:w-10/12 lg:w-9/12">
<ul class="container s:mx-auto flex-row hidden flex-wrap lg:flex">
{% for post in site.posts %} {% include post-preview.html %} {% endfor %}
{% for post in site.posts %} {% if post.category == "research" %} {%
include post-preview.html %} {% endif %} {% endfor %}
</ul>
<ul
class="
@ -72,8 +73,8 @@ layout: default
lg:hidden
"
>
{% for post in site.posts limit:8 %} {% include post-preview.html %} {%
endfor %}
{% for post in site.posts limit:8 %} {% if post.category == "research" %}
{% include post-preview.html %} {% endif %} {% endfor %}
</ul>
</div>
</section>
@ -89,8 +90,8 @@ layout: default
<div class="info-block w-full sm:w-10/12 lg:w-9/12">
<ul class="container s:mx-auto flex flex-col flex-wrap sm:flex-row">
{% for post in site.posts limit:4%} {% include post-preview.html %} {%
endfor %}
{% for post in site.posts %} {% if post.category == "problem" %} {%
include post-preview.html %} {% endif %} {% endfor %}
</ul>
</div>
</section>

22
_posts/2019-07-19-p2p-data-sync-for-mobile.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "P2P Data Sync for Mobile"
title: "P2P Data Sync for Mobile"
date: 2019-07-19 12:00:00 +0800
name: "P2P Data Sync for Mobile"
title: "P2P Data Sync for Mobile"
date: 2019-07-19 12:00:00 +0800
author: oskarth
published: true
permalink: /p2p-data-sync-for-mobile
categories: research
category: research
summary: A research log. Reliable and decentralized, pick two.
image: /assets/img/mvds_interactive.png
---
@ -22,14 +22,17 @@ There are many synchronization protocols out there and I won't go into detail of
Why do we want to do p2p sync for mobilephones in the first place? There are three components to that question. One is on the value of decentralization and peer-to-peer, the second is on why we'd want to reliably sync data at all, and finally why mobilephones and other resource restricted devices.
### Why p2p?
For decentralization and p2p, there are both technical and social/philosophical reasons. Technically, having a user-run network means it can scale with the number of users. Data locality is also improved if you query data that's close to you, similar to distributed CDNs. The throughput is also improved if there are more places to get data from.
Socially and philosophically, there are several ways to think about it. Open and decentralized networks also relate to the idea of open standards, i.e. compare the longevity of AOL with IRC or Bittorrent. One is run by a company and is shut down as soon as it stops being profitable, the others live on. Additionally increasingly control of data and infrastructure is becoming a liability. By having a network with no one in control, everyone is. It's ultimately a form of democratization, more similar to organic social structures pre Big Internet companies. This leads to properties such as censorship resistance and coercion resistance, where we limit the impact a 3rd party might have a voluntary interaction between individuals or a group of people. Examples of this are plentiful in the world of Facebook, Youtube, Twitter and WeChat.
### Why reliably sync data?
At risk of stating the obvious, reliably syncing data is a requirement for many problem domains. You don't get this by default in a p2p world, as it is unreliable with nodes permissionslessly join and leave the network. In some cases you can get away with only ephemeral data, but usually you want some kind of guarantees. This is a must for reliable group chat experience, for example, where messages are expected to arrive in a timely fashion and in some reasonable order. The same is true for messages there represent financial transactions, and so on.
### Why mobilephones?
Most devices people use daily are mobile phones. It's important to provide the same or at least similar guarantees to more traditional p2p nodes that might run on a desktop computer or computer. The alternative is to rely on gateways, which shares many of the drawbacks of centralized control and prone to censorship, control and surveillence.
More generally, resource restricted devices can differ in their capabilities. One example is smartphones, but others are: desktop, routers, Raspberry PIs, POS systems, and so on. The number and diversity of devices are exploding, and it's useful to be able to leverage this for various types of infrastructure. The alternative is to centralize on big cloud providers, which also lends itself to lack of democratization and censorship, etc.
@ -90,18 +93,19 @@ Each failed attempt implies another retransmission. That means we have `(1/0.1)^
The problem above hints at the requirements 3 and 4 above. While we did get reliable syncing (requirement 1), it came at a big cost.
There are a few ways of getting around this issue. One is having a *store and forward* model, where some intermediary node picks up (encrypted) messages and forwards them to the recipient. This is what we have in production right now at Status.
There are a few ways of getting around this issue. One is having a _store and forward_ model, where some intermediary node picks up (encrypted) messages and forwards them to the recipient. This is what we have in production right now at Status.
Another, arguably more pure and robust, way is having a *remote log*, where the actual data is spread over some decentralized storage layer, and you have a mutable reference to find the latest messages, similar to DNS.
Another, arguably more pure and robust, way is having a _remote log_, where the actual data is spread over some decentralized storage layer, and you have a mutable reference to find the latest messages, similar to DNS.
What they both have in common is that they act as a sort of highly-available cache to smooth over the non-overlapping connection windows between two endpoints. Neither of them are *required* to get reliable data transmission.
What they both have in common is that they act as a sort of highly-available cache to smooth over the non-overlapping connection windows between two endpoints. Neither of them are _required_ to get reliable data transmission.
### Basic calculations for bandwidth multiplier
While we do want better simulations, and this is a work in progress, we can also look at the above scenarios using some basic calculations. This allows us to build a better intuition and reason about the problem without having to write code. Let's start with some assumptions:
- two nodes exchanging a single message in batch mode
- 10% uniformly random uptime for each node
- in HA cache case, 100% uptime of a piece of infrastructure C
- in HA cache case, 100% uptime of a piece of infrastructure C
- retransmission every epoch (with constant or exponential backoff)
- only looking at average (p50) case
@ -116,7 +120,7 @@ A <- ack B (10% chance of arrival)
With a constant backoff, A will send messages at epoch `1, 2, 3, ...`. With exponential backoff and a multiplier of 2, this would be `1, 2, 4, 8, ...`. Let's assume constant backoff for now, as this is what will influence the success rate and thus the bandwidth multiplier.
There's a difference between *time to receive* and *time to stop sending*. Assuming each send attempt is independent, it takes on average 10 epochs for A's message to arrive with B. Furthermore:
There's a difference between _time to receive_ and _time to stop sending_. Assuming each send attempt is independent, it takes on average 10 epochs for A's message to arrive with B. Furthermore:
1. A will send messages until it receives an ACK.
2. B will send ACK if it receives a message.

41
_posts/2019-08-02-vac-overview.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "Vac - A Rough Overview"
title: "Vac - A Rough Overview"
date: 2019-08-02 12:00:00 +0800
name: "Vac - A Rough Overview"
title: "Vac - A Rough Overview"
date: 2019-08-02 12:00:00 +0800
author: oskarth
published: true
permalink: /vac-overview
categories: research
category: research
summary: Vac is a modular peer-to-peer messaging stack, with a focus on secure messaging. Overview of terms, stack and open problems.
---
@ -14,31 +14,31 @@ Vac is a **modular peer-to-peer messaging stack, with a focus on secure messagin
## Basic terms
*messaging stack*. While the initial focus is on [data sync](https://vac.dev/p2p-data-sync-for-mobile), we are concerned with all layers in the stack. That means all the way from underlying transports, p2p overlays and routing, to initial trust establishment and semantics for things like group chat. The ultimate goal is to give application developers the tools they need to provide secure messaging for their users, so they can focus on their domain expertise.
_messaging stack_. While the initial focus is on [data sync](https://vac.dev/p2p-data-sync-for-mobile), we are concerned with all layers in the stack. That means all the way from underlying transports, p2p overlays and routing, to initial trust establishment and semantics for things like group chat. The ultimate goal is to give application developers the tools they need to provide secure messaging for their users, so they can focus on their domain expertise.
*modular*. Unlike many other secure messaging applications, our goal is not to have a tightly coupled set of protocols, nor is it to reinvent the wheel. Instead, we aim to provide options at each layer in the stack, and build on the shoulders of giants, putting a premimum on interoperability. It's similar in philosophy to projects such as [libp2p](https://libp2p.io/) or [Substrate](https://www.parity.io/substrate/) in that regard. Each choice comes with different trade-offs, and these look different for different applications.
_modular_. Unlike many other secure messaging applications, our goal is not to have a tightly coupled set of protocols, nor is it to reinvent the wheel. Instead, we aim to provide options at each layer in the stack, and build on the shoulders of giants, putting a premimum on interoperability. It's similar in philosophy to projects such as [libp2p](https://libp2p.io/) or [Substrate](https://www.parity.io/substrate/) in that regard. Each choice comes with different trade-offs, and these look different for different applications.
*peer-to-peer*. The protocols we work on are pure p2p, and aim to minimize centralization. This too is in opposition to many initiatives in the secure messaging space.
_peer-to-peer_. The protocols we work on are pure p2p, and aim to minimize centralization. This too is in opposition to many initiatives in the secure messaging space.
*messaging*. By messaging we mean messaging in a generalized sense. This includes both human to human communication, as well machine to machine communication. By messaging we also mean something more fundamental than text messages, we also include things like transactions (state channels, etc) under this moniker.
_messaging_. By messaging we mean messaging in a generalized sense. This includes both human to human communication, as well machine to machine communication. By messaging we also mean something more fundamental than text messages, we also include things like transactions (state channels, etc) under this moniker.
*secure messaging*. Outside of traditional notions of secure messaging, such as ensuring end to end encryption, forward secrecy, avoiding MITM-attacks, etc, we are also concerned with two other forms of secure messaging. We call these *private messaging* and *censorship-resistance*. Private messaging means viewing privacy as a security property, with all that entails. Censorship resistance ties into being p2p, but also in terms of allowing for transports and overlays that can't easily be censored by port blocking, traffic analysis, and similar.
_secure messaging_. Outside of traditional notions of secure messaging, such as ensuring end to end encryption, forward secrecy, avoiding MITM-attacks, etc, we are also concerned with two other forms of secure messaging. We call these _private messaging_ and _censorship-resistance_. Private messaging means viewing privacy as a security property, with all that entails. Censorship resistance ties into being p2p, but also in terms of allowing for transports and overlays that can't easily be censored by port blocking, traffic analysis, and similar.
*Vāc*. Is a Vedic goddess of speech. It also hints at being a vaccine.
_Vāc_. Is a Vedic goddess of speech. It also hints at being a vaccine.
## Protocol stack
What does this stack look like? We take inspiration from [core](https://tools.ietf.org/html/rfc793) [internet architecture](https://www.ietf.org/rfc/rfc1122.txt), existing [survey work](http://cacr.uwaterloo.ca/techreports/2015/cacr2015-02.pdf) and other [efforts](https://code.briarproject.org/briar/briar/wikis/A-Quick-Overview-of-the-Protocol-Stack) that have been done to decompose the problem into orthogonal pieces. Each layer provides their own set of properties and only interact with the layers it is adjacent to. Note that this is a rough sketch.
| Layer / Protocol | Purpose | Examples |
|-------------------|-----------------------------------|----------------------|
| Application layer | End user semantics | 1:1 chat, group chat |
| Data Sync | Data consistency | MVDS, BSP |
| Secure Transport | Confidentiality, PFS, etc | Double Ratchet, MLS |
| Transport Privacy | Transport and metadata protection | Whisper, Tor, Mixnet |
| P2P Overlay | Overlay routing, NAT traversal | devp2p, libp2p |
| | |
| Trust Establishment | Establishing end-to-end trust | TOFU, web of trust |
| Layer / Protocol | Purpose | Examples |
| ------------------- | --------------------------------- | -------------------- |
| Application layer | End user semantics | 1:1 chat, group chat |
| Data Sync | Data consistency | MVDS, BSP |
| Secure Transport | Confidentiality, PFS, etc | Double Ratchet, MLS |
| Transport Privacy | Transport and metadata protection | Whisper, Tor, Mixnet |
| P2P Overlay | Overlay routing, NAT traversal | devp2p, libp2p |
| | |
| Trust Establishment | Establishing end-to-end trust | TOFU, web of trust |
As an example, end user semantics such as group chat or moderation capabilities can largely work regardless of specific choices further down the stack. Similarly, using a mesh network or Tor doesn't impact the use of Double Ratchet at the Secure Transport layer.
@ -51,11 +51,13 @@ In terms of specific properties and trade-offs at each layer, we'll go deeper do
With all the pieces involved, this is quite an undertaking. Luckily, a lot of pieces are already in place and can be either incorporated as-is or iterated on. In terms of medium and long term, here's a rough sketch of priorities and open problems.
1. **Better data sync.** While the current [MVDS](https://specs.vac.dev/specs/mvds.html) works, it is lacking in a few areas:
- Lack of remote log for mostly-offline offline devices
- Better scalability for multi-user chat contexts
- Better usability in terms of application layer usage and supporting more transports
2. **Better transport layer support.** Currently MVDS runs primarily over Whisper, which has a few issues:
- scalability, being able to run with many nodes
- spam-resistance, proof of work is a poor mechanism for heterogeneous devices
- no incentivized infrastructure, leading to centralized choke points
@ -67,7 +69,6 @@ In addition to these most immediate concerns, there are other open problems. Som
4. **Incentivized and spam-resistant messaging.** Reasons to run infrastructure and not relying on altruistic nodes. For spam resistance, in p2p multicast spam is a big attack vector due to amplification. There are a few interesting directions here, such as EigenTrust, proof of burn with micropayments, and leveraging zero-knowledge proofs.
5. **Strong privacy guarantees at transport privacy layer**. More rigorous privacy guarantees and explicit trade-offs for metadata protection. Includes Mixnet.
6. **Censorship-resistant and robust P2P overlay**. NAT traversal; running in the browser; mesh networks; pluggable transports for traffic obfuscation.
7. **Scalable and decentralized secure conversational security.** Strong security guarantees such as forward secrecy, post compromise security, for large group chats. Includes projects such MLS and extending Double Ratchet.

60
_posts/2019-10-04-remote-log.md
View File

@ -1,21 +1,21 @@
---
layout: post
name: "P2P Data Sync with a Remote Log"
title: "P2P Data Sync with a Remote Log"
date: 2019-10-04 12:00:00 +0800
name: "P2P Data Sync with a Remote Log"
title: "P2P Data Sync with a Remote Log"
date: 2019-10-04 12:00:00 +0800
author: oskarth
published: true
permalink: /remote-log
categories: research
category: research
summary: A research log. Asynchronous P2P messaging? Remote logs to the rescue!
image: /assets/img/remote-log.png
---
A big problem when doing end-to-end data sync between mobile nodes is that most devices are offline most of the time. With a naive approach, you quickly run into issues of 'ping-pong' behavior, where messages have to be constantly retransmitted. We saw some basic calculations of what this bandwidth multiplier looks like in a [previous post](https://vac.dev/p2p-data-sync-for-mobile).
While you could do some background processing, this is really battery-draining, and on iOS these capabilities are limited. A better approach instead is to loosen the constraint that two nodes need to be online at the same time. How do we do this? There are two main approaches, one is the *store and forward model*, and the other is a *remote log*.
While you could do some background processing, this is really battery-draining, and on iOS these capabilities are limited. A better approach instead is to loosen the constraint that two nodes need to be online at the same time. How do we do this? There are two main approaches, one is the _store and forward model_, and the other is a _remote log_.
In the *store and forward* model, we use an intermediate node that forward messages on behalf of the recipient. In the *remote log* model, you instead replicate the data onto some decentralized storage, and have a mutable reference to the latest state, similar to DNS. While both work, the latter is somewhat more elegant and "pure", as it has less strict requirements of an individual node's uptime. Both act as a highly-available cache to smoothen over non-overlapping connection windows between endpoints.
In the _store and forward_ model, we use an intermediate node that forward messages on behalf of the recipient. In the _remote log_ model, you instead replicate the data onto some decentralized storage, and have a mutable reference to the latest state, similar to DNS. While both work, the latter is somewhat more elegant and "pure", as it has less strict requirements of an individual node's uptime. Both act as a highly-available cache to smoothen over non-overlapping connection windows between endpoints.
In this post we are going to describe how such a remote log schema could work. Specifically, how it enhances p2p data sync and takes care of the [following requirements](https://vac.dev/p2p-data-sync-for-mobile):
@ -31,15 +31,15 @@ In this post we are going to describe how such a remote log schema could work. S
A remote log is a replication of a local log. This means a node can read data from a node that is offline.
The spec is in an early draft stage and can be found [here](https://github.com/vacp2p/specs/pull/16). A very basic [spike](https://en.wikipedia.org/wiki/Spike_(software_development)) / proof-of-concept can be found [here](https://github.com/vacp2p/research/tree/master/remote_log).
The spec is in an early draft stage and can be found [here](https://github.com/vacp2p/specs/pull/16). A very basic [spike](<https://en.wikipedia.org/wiki/Spike_(software_development)>) / proof-of-concept can be found [here](https://github.com/vacp2p/research/tree/master/remote_log).
### Definitions
| Term | Definition |
| ----------- | -------------------------------------------------------------------------------------- |
| CAS | Content-addressed storage. Stores data that can be addressed by its hash. |
| NS | Name system. Associates mutable data to a name. |
| Remote log | Replication of a local log at a different location. |
| Term | Definition |
| ---------- | ------------------------------------------------------------------------- |
| CAS | Content-addressed storage. Stores data that can be addressed by its hash. |
| NS | Name system. Associates mutable data to a name. |
| Remote log | Replication of a local log at a different location. |
### Roles
@ -47,10 +47,10 @@ There are four fundamental roles:
1. Alice
2. Bob
2. Name system (NS)
3. Content-addressed storage (CAS)
3. Name system (NS)
4. Content-addressed storage (CAS)
The *remote log* is the data format of what is stored in the name system.
The _remote log_ is the data format of what is stored in the name system.
"Bob" can represent anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS.
@ -75,22 +75,22 @@ The remote log lets receiving nodes know what data they are missing. Depending o
A remote log is simply a mapping from message identifiers to their corresponding address in a CAS:
| Message Identifier (H1) | CAS Hash (H2) |
| ---------------- |---------------|
| H1_3 | H2_3 |
| H1_2 | H2_2 |
| H1_1 | H2_1 |
| | |
| *address to next page* |
| ----------------------- | ------------- |
| H1_3 | H2_3 |
| H1_2 | H2_2 |
| H1_1 | H2_1 |
| | |
| _address to next page_ |
The numbers here corresponds to messages. Optionally, the content itself can be included, just like it normally would be sent over the wire. This bypasses the need for a dedicated CAS and additional round-trips, with a trade-off in bandwidth usage.
| Message Identifier (H1) | Content |
| ---------------- |---------------|
| H1_3 | C3 |
| H1_2 | C2 |
| H1_1 | C1 |
| | |
| *address to next page* |
| Message Identifier (H1) | Content |
| ----------------------- | ------- |
| H1_3 | C3 |
| H1_2 | C2 |
| H1_1 | C1 |
| | |
| _address to next page_ |
Both patterns can be used in parallel, e,g. by storing the last `k` messages directly and use CAS pointers for the rest. Together with the `next_page` page semantics, this gives users flexibility in terms of bandwidth and latency/indirection, all the way from a simple linked list to a fully replicated log. The latter is useful for things like backups on durable storage.
@ -103,13 +103,13 @@ Both patterns can be used in parallel, e,g. by storing the last `k` messages dir
The spec is still in an early draft stage, so it is expected to change. Same with the proof of concept. More work is needed on getting a fully featured proof of concept with specific CAS and NAS instances. E.g. Swarm and Swarm Feeds, or IPFS and IPNS, or something else.
For data sync in general:
- Make consistency guarantees more explicit for app developers with support for sequence numbers and DAGs, as well as the ability to send non-synced messages. E.g. ephemeral typing notifications, linear/sequential history and casual consistency/DAG history
- Make consistency guarantees more explicit for app developers with support for sequence numbers and DAGs, as well as the ability to send non-synced messages. E.g. ephemeral typing notifications, linear/sequential history and casual consistency/DAG history
- Better semantics and scalability for multi-user sync contexts, e.g. CRDTs and joining multiple logs together
- Better usability in terms of application layer usage (data sync clients) and supporting more transports
---
PS1. Thanks everyone who submitted great [logo proposals](https://explorer.bounties.network/bounty/3389) for Vac!
PPS2. Next week on October 10th decanus and I will be presenting Vac at [Devcon](https://devcon.org/agenda), come say hi :)

33
_posts/2019-11-08-feasibility-semaphore-rate-limiting-zksnarks.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "Feasibility Study: Semaphore rate limiting through zkSNARKs"
title: "Feasibility Study: Semaphore rate limiting through zkSNARKs"
date: 2019-11-08 12:00:00 +0800
name: "Feasibility Study: Semaphore rate limiting through zkSNARKs"
title: "Feasibility Study: Semaphore rate limiting through zkSNARKs"
date: 2019-11-08 12:00:00 +0800
author: oskarth
published: true
permalink: /feasibility-semaphore-rate-limiting-zksnarks
categories: research
category: research
summary: A research log. Zero knowledge signaling as a rate limiting mechanism to prevent spam in p2p networks.
image: /assets/img/peacock-signaling.jpg
discuss: https://forum.vac.dev/t/discussion-feasibility-study-semaphore-rate-limiting-through-zksnarks/21
@ -33,13 +33,13 @@ In addition to above, there are a lot of related problems that share similaritie
## Basic terminology
A *zero-knowledge proof* allows a *prover* to show a *verifier* that they know something, without revealing what that something is. This means you can do trust-minimized computation that is also privacy preserving. As a basic example, instead of showing your ID when going to a bar you simply give them a proof that you are over 18, without showing the doorman your id.
A _zero-knowledge proof_ allows a _prover_ to show a _verifier_ that they know something, without revealing what that something is. This means you can do trust-minimized computation that is also privacy preserving. As a basic example, instead of showing your ID when going to a bar you simply give them a proof that you are over 18, without showing the doorman your id.
*zkSNARKs* is a form of zero-knowledge proofs. There are many types of zero-knowledge proofs, and the field is evolving rapidly. They come with various trade-offs in terms of things such as: trusted setup, cryptographic assumptions, proof/verification key size, proof/verification time, proof size, etc. See section below for more.
_zkSNARKs_ is a form of zero-knowledge proofs. There are many types of zero-knowledge proofs, and the field is evolving rapidly. They come with various trade-offs in terms of things such as: trusted setup, cryptographic assumptions, proof/verification key size, proof/verification time, proof size, etc. See section below for more.
*Semaphore* is a framework/library/construct on top of zkSNARks. It allows for zero-knowledge signaling, specifically on top of Ethereum. This means an approved user can broadcast some arbitrary string without revealing their identity, given some specific constraints. An approved user is someone who has been added to a certain merkle tree. See [current Github home](https://github.com/kobigurk/semaphore) for more.
_Semaphore_ is a framework/library/construct on top of zkSNARks. It allows for zero-knowledge signaling, specifically on top of Ethereum. This means an approved user can broadcast some arbitrary string without revealing their identity, given some specific constraints. An approved user is someone who has been added to a certain merkle tree. See [current Github home](https://github.com/kobigurk/semaphore) for more.
*Circom* is a DSL for writing arithmetic circuits that can be used in zkSNARKs, similar to how you might write a NAND gate. See [Github](https://github.com/iden3/circom) for more.
_Circom_ is a DSL for writing arithmetic circuits that can be used in zkSNARKs, similar to how you might write a NAND gate. See [Github](https://github.com/iden3/circom) for more.
## Basic flow
@ -47,28 +47,23 @@ We start with a private voting example, and then extend it to the slashable rate
1. A user registers an identity (arbitrary keypair), along with a small fee, to a smart contract. This adds them to a merkle tree and allows them to prove that they are member of that group, without revealing who they are.
2. When a user wants to send a message, they compute a zero-knowledge proof. This ensures certain invariants, have some *public outputs*, and can be verified by anyone (including a smart contract).
2. When a user wants to send a message, they compute a zero-knowledge proof. This ensures certain invariants, have some _public outputs_, and can be verified by anyone (including a smart contract).
3. Any node can verify the proof, including smart contracts on chain (as of Byzantinum HF). Additionally, a node can have rules for the public output. In the case of voting, one such rule is that a specific output hash has to be equal to some predefined value, such as "2020-01-01 vote on Foo Bar for president".
4. Because of how the proof is constructed, and the rules around output values, this ensures that: a user is part of the approved set of voters and that a user can only vote once.
5. As a consequence of above, we have a system where registered users can only vote once, no one can see who voted for what, and this can all be proven and verified.
### Rate limiting example
In the case of rate limiting, we do want nodes to send multiple messages. This changes step 3-5 above somewhat.
*NOTE: It is a bit more involved than this, and if we precompute proofs the flow might look a bit different. But the general idea is the same*.
_NOTE: It is a bit more involved than this, and if we precompute proofs the flow might look a bit different. But the general idea is the same_.
1. Instead of having a rule that you can only vote once, we have a rule that you can only send a message per epoch. Epoch here can be every second, as defined by UTC date time +-20s.
2. Additionally, if a users sends more than one message per epoch, one of the public outputs is a random share of a private key. Using Shamir's Secret Sharing (similar to a multisig) and 2/3 key share as an example threshold: in the normal case only 1/3 private keys is revealed, which is insufficient to have access. In the case where two messages are sent in an epoch, probabilistically 2/3 shares is sufficient to have access to the key (unless you get the same random share of the key).
3. This means any untrusted user who detects a spamming user, can use it to access their private key corresponding to funds in the contract, and thus slash them.
4. As a consequence of above, we have a system where registered users can only messages X times per epoch, and no one can see who is sending what messages. Additionally, if a user is violating the above rate limit, they can be punished and any user can profit from it.
### Briefly on scope of 'approved users'
In the case of an application like Status, this construct can either be a global StatusNetwork group, or one per chat, or network, etc. It can be applied both at the network and user level. There are no specific limitations on where or who deploys this, and it is thus more of a UX consideration.
@ -119,7 +114,7 @@ See [details](https://github.com/vacp2p/research/issues/11).
#### Licensing issues
Currently Circom [uses a GPL license](https://github.com/iden3/circom/blob/master/COPYING ), which can get tricky when it comes to the App Store etc.
Currently Circom [uses a GPL license](https://github.com/iden3/circom/blob/master/COPYING), which can get tricky when it comes to the App Store etc.
See [details](https://github.com/vacp2p/research/issues/12).
@ -149,4 +144,4 @@ For example, this might also include leveraging largely ready made solutions suc
Thanks to Barry Whitehat for patient explanation and pointers. Thanks to WJ for helping with runtime issues.
*Peacock header image from [Tonos](https://en.wikipedia.org/wiki/File:Flickr_-_lo.tangelini_-_Tonos_(1).jpg).*
_Peacock header image from [Tonos](<https://en.wikipedia.org/wiki/File:Flickr_-_lo.tangelini_-_Tonos_(1).jpg>)._

12
_posts/2019-12-03-fixing-whisper-with-waku.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "Fixing Whisper with Waku"
title: "Fixing Whisper with Waku"
date: 2019-12-03 12:00:00 +0800
name: "Fixing Whisper with Waku"
title: "Fixing Whisper with Waku"
date: 2019-12-03 12:00:00 +0800
author: oskarth
published: true
permalink: /fixing-whisper-with-waku
categories: research
category: research
summary: A research log. Why Whisper doesn't scale and how to fix it.
image: /assets/img/whisper_scalability.png
discuss: https://forum.vac.dev/t/discussion-fixing-whisper-with-waku/27
@ -41,7 +41,7 @@ In this post, we'll focus on the first problem, which is scalability through ban
## Whisper theoretical scalability model
*(Feel free to skip this section if you want to get right to the results).*
_(Feel free to skip this section if you want to get right to the results)._
There's widespread implicit knowledge that Whisper "doesn't scale", but it is less understood exactly why. This theoretical model attempts to encode some characteristics of it. Specifically for use case such as one by Status (see [Status Whisper usage
spec](https://github.com/status-im/specs/blob/master/status-whisper-usage-spec.md)).
@ -289,6 +289,6 @@ iterative work with results on the order of weeks.
In short, we have a [Waku version 0 spec up](https://specs.vac.dev/specs/waku.html) as well as a [PoC](https://github.com/status-im/nim-eth/pull/120) for backwards compatibility. In the coming weeks, we are going to solidify the specs, get a more fully featured PoC for [Waku mode](https://github.com/status-im/nim-eth/pull/114). See [rough roadmap](https://github.com/vacp2p/pm/issues/5), [project board](https://github.com/orgs/vacp2p/projects/2) and progress thread on the [Vac forum](https://forum.vac.dev/t/waku-project-and-progress/24).
The spec has been rewrittten for clarity, with ABNF grammar and less ambiguous language. The spec also incorporates several previously [ad hoc implemented features](https://specs.vac.dev/specs/waku.html#additional-capabilities), such as light nodes and mailserver/client support. This has already caught a few incompatibilities between the `geth` (Go), `status/whisper` (Go) and `nim-eth` (Nim) versions, specifically around light node usage and the handshake.
The spec has been rewrittten for clarity, with ABNF grammar and less ambiguous language. The spec also incorporates several previously [ad hoc implemented features](https://specs.vac.dev/specs/waku.html#additional-capabilities), such as light nodes and mailserver/client support. This has already caught a few incompatibilities between the `geth` (Go), `status/whisper` (Go) and `nim-eth` (Nim) versions, specifically around light node usage and the handshake.
If you are interested in this effort, please check out [our forum](https://forum.vac.dev/) for questions, comments and proposals. We already have some discussion for better [spam protection](https://forum.vac.dev/t/stake-priority-based-queuing/26) (see [previous post](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks) for a more complex but privacy-preserving proposal), something that is likely going to be addressed in future versions of Waku, along with many other fixes and enhancement.

52
_posts/2020-02-14-waku-update.md
View File

@ -1,18 +1,18 @@
---
layout: post
name: "Waku Update"
title: "Waku Update"
date: 2020-02-14 12:00:00 +0800
name: "Waku Update"
title: "Waku Update"
date: 2020-02-14 12:00:00 +0800
author: oskarth
published: true
permalink: /waku-update
categories: research
category: research
summary: A research log. What's the current state of Waku? How many users does it support? What are the bottlenecks? What's next?
image: /assets/img/waku_infrastructure_sky.jpg
discuss: https://forum.vac.dev/t/waku-update-where-are-we-at/34
---
Waku is our fork of Whisper where we address the shortcomings of Whisper in an iterative manner. We've seen a in [previous post](https://vac.dev/fixing-whisper-with-waku) that Whisper doesn't scale, and why. In this post we'll talk about what the current state of Waku is, how many users it can support, and future plans.
Waku is our fork of Whisper where we address the shortcomings of Whisper in an iterative manner. We've seen a in [previous post](https://vac.dev/fixing-whisper-with-waku) that Whisper doesn't scale, and why. In this post we'll talk about what the current state of Waku is, how many users it can support, and future plans.
## Current state
@ -51,12 +51,13 @@ We have a [simulation](https://github.com/status-im/nimbus/tree/master/waku#test
This is our current understanding of how many users a network running Waku can support. Specifically in the context of the Status chat app, since that's the most immediate consumer of Waku. It should generalize fairly well to most deployments.
**tl;dr (for Status app):**
- beta: 100 DAU
- v1: 1k DAU
- v1.1 (waku only): 10k DAU (up to x10 with deployment hotfixes)
- v1.2 (waku+dns): 100k DAU (can optionally be folded into v1.1)
*Assuming 10 concurrent users = 100 DAU. Estimate uncertainty increases for each order of magnitude until real-world data is observed.*
_Assuming 10 concurrent users = 100 DAU. Estimate uncertainty increases for each order of magnitude until real-world data is observed._
As far as we know right now, these are the bottlenecks we have:
@ -83,30 +84,31 @@ For light node, bloom filter is set to almost 10% false positive (bloom filter:
**Star network:**
| Description | Peers | Valid | Invalid |
|-----------------|-------|-------|---------|
| Master node | 7 | 10001 | 0 |
| Full node 1 | 3 | 10001 | 0 |
| Full node 2 | 1 | 10001 | 0 |
| Full node 3 | 1 | 10001 | 0 |
| Full node 4 | 1 | 10001 | 0 |
| Full node 5 | 1 | 10001 | 0 |
| Light node | 2 | 815 | 0 |
| Waku light node | 2 | 1 | 0 |
| --------------- | ----- | ----- | ------- |
| Master node | 7 | 10001 | 0 |
| Full node 1 | 3 | 10001 | 0 |
| Full node 2 | 1 | 10001 | 0 |
| Full node 3 | 1 | 10001 | 0 |
| Full node 4 | 1 | 10001 | 0 |
| Full node 5 | 1 | 10001 | 0 |
| Light node | 2 | 815 | 0 |
| Waku light node | 2 | 1 | 0 |
**Full mesh:**
| Description | Peers | Valid | Invalid |
|-----------------|-------|-------|---------|
| Full node 0 | 7 | 10001 | 20676 |
| Full node 1 | 7 | 10001 | 9554 |
| Full node 2 | 5 | 10001 | 23304 |
| Full node 3 | 5 | 10001 | 11983 |
| Full node 4 | 5 | 10001 | 24425 |
| Full node 5 | 5 | 10001 | 23472 |
| Light node | 2 | 803 | 803 |
| Waku light node | 2 | 1 | 1 |
| --------------- | ----- | ----- | ------- |
| Full node 0 | 7 | 10001 | 20676 |
| Full node 1 | 7 | 10001 | 9554 |
| Full node 2 | 5 | 10001 | 23304 |
| Full node 3 | 5 | 10001 | 11983 |
| Full node 4 | 5 | 10001 | 24425 |
| Full node 5 | 5 | 10001 | 23472 |
| Light node | 2 | 803 | 803 |
| Waku light node | 2 | 1 | 1 |
Things to note:
- Whisper light node with ~10% false positive gets ~10% of total traffic
- Waku light node gets ~1000x less envelopes than Whisper light node
- Full mesh results in a lot more duplicate messages, expect for Waku light node
@ -141,4 +143,4 @@ For the third bottleneck, a likely candidate for fixing this is Kademlia routing
## Acknowledgements
*Image from "caged sky" by mh.xbhd.org is licensed under CC BY 2.0 (https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e)*
_Image from "caged sky" by mh.xbhd.org is licensed under CC BY 2.0 (https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e)_

23
_posts/2020-02-7-dns-based-discovery.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "DNS Based Discovery"
title: "DNS Based Discovery"
date: 2020-02-7 12:00:00 +0100
name: "DNS Based Discovery"
title: "DNS Based Discovery"
date: 2020-02-7 12:00:00 +0100
author: dean
published: true
permalink: /dns-based-discovery
categories: research
category: research
summary: A look at EIP-1459 and the benefits of DNS based discovery.
---
@ -16,11 +16,11 @@ Another issue is how we do the initial bootstrapping. In other words, how does a
In summary, we have a static list that provides us with nodes we can connect to which then allows us to discover the rest of the network using something like Kademlia. But what we need is something that can easily be mutated, guarantees a certain amount of security, and is efficient for resource restricted devices. Ideally our solution would also be robust and scalable.
How do we do this?
How do we do this?
[EIP 1459: Node Discovery via DNS](https://eips.ethereum.org/EIPS/eip-1459), which is one of the strategies we are using for discovering waku nodes. [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459) is a DNS-based discovery protocol that stores [merkle trees](https://en.wikipedia.org/wiki/Merkle_tree) in DNS records which contain connection information for nodes.
*Waku is our fork of Whisper. Oskar recently wrote an [entire post](https://vac.dev/fixing-whisper-with-waku) explaining it. In short, Waku is our method of fixing the shortcomings of Whisper in a more iterative fashion. You can find the specification [here](https://specs.vac.dev/specs/waku/waku.html)*
_Waku is our fork of Whisper. Oskar recently wrote an [entire post](https://vac.dev/fixing-whisper-with-waku) explaining it. In short, Waku is our method of fixing the shortcomings of Whisper in a more iterative fashion. You can find the specification [here](https://specs.vac.dev/specs/waku/waku.html)_
DNS-based methods for bootstrapping p2p networks are quite popular. Even Bitcoin uses it, but it uses a concept called DNS seeds, which are just DNS servers that are configured to return a list of randomly selected nodes from the network upon being queried. This means that although these seeds are hardcoded in the client, the IP addresses of actual nodes do not have to be.
@ -64,12 +64,13 @@ There are multiple reasons, but the main one is **security**. In the Bitcoin exa
Let's dive into this. Firstly, a client that is using these DNS records for discovery must know the public key corresponding to the private key controlled by the entity creating the list. This is because the entire list is signed using a secp256k1 private key, giving the client the ability to authenticate the list and know that it has not been tampered with by some external party.
So that already makes this a lot safer than the method Bitcoin uses. But how are these lists even stored? As previously stated they are stored using **merkle trees** as follows:
- The root of the tree is stored in a [`TXT` record](https://simpledns.plus/help/txt-records), this record contains the tree's root hash, a sequence number which is incremented every time the tree is updated and a signature as stated above.
Additionally, there is also a root hash to a second tree called a **link tree**, it contains the information to different lists. This link tree allows us to delegate trust and build a graph of multiple merkle trees stored across multiple DNS names.
The sequence number ensures that an attacker cannot replace a tree with an older version because when a client reads the tree, they should ensure that the sequence number is greater than the last synchronized version.
- The root of the tree is stored in a [`TXT` record](https://simpledns.plus/help/txt-records), this record contains the tree's root hash, a sequence number which is incremented every time the tree is updated and a signature as stated above.
Additionally, there is also a root hash to a second tree called a **link tree**, it contains the information to different lists. This link tree allows us to delegate trust and build a graph of multiple merkle trees stored across multiple DNS names.
The sequence number ensures that an attacker cannot replace a tree with an older version because when a client reads the tree, they should ensure that the sequence number is greater than the last synchronized version.
- Using the root hash for the tree, we can find the merkle tree's first branch, the branch is also stored in a `TXT` record. The branch record contains all the hashes of the branch's leafs.
- Once a client starts reading all the leafs, they can find one of two things: either a new branch record leading them further down the tree or an Ethereum Name Records (ENR) which means they now have the address of a node to connect to! To learn more about ethereum node records you can have a look at [EIP-778](https://eips.ethereum.org/EIPS/eip-778), or read a short blog post I wrote explaining them [here](https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/#enr).

28
_posts/2020-04-16-wechat-replacement-need.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "What Would a WeChat Replacement Need?"
title: "What Would a WeChat Replacement Need?"
date: 2020-04-16 12:00:00 +0800
name: "What Would a WeChat Replacement Need?"
title: "What Would a WeChat Replacement Need?"
date: 2020-04-16 12:00:00 +0800
author: oskarth
published: true
permalink: /wechat-replacement-need
categories: research
category: research
summary: What would a self-sovereign, private, censorship-resistant and open alternative to WeChat look like?
image: /assets/img/tianstatue.jpg
discuss: https://forum.vac.dev/t/discussion-what-would-a-wechat-replacement-need/42
@ -84,11 +84,12 @@ We want something that is self-sovereign, private, censorship-resistant and open
**Open source**. Related to above, where we must be able to inspect the software to know that it functions as advertised and hasn't been compromised, e.g. by uploading private data to a third party.
Some of these are graded and a bit subtle, i.e.:
- Censorship resistance would ideally be able to absorb Internet shutdowns. This would require an extensive MANET/meshnet infrastructure, which while desirable, requires a lot of challenges to be overcome to be feasible.
- Privacy would ideally make all actions (optionally) totally anoymous, though this may incur undue costs on bandwidth and latency, which impacts user experience.
- Decentralization, certain topologies, such as DHTs, are efficient and quite decentralized but still have some centralized aspects, which makes it attackable in various ways. Ditto for blockchains compared with bearer instruments which requires some coordinating infrastructure, compared with naturally occuring assets such as precious metals.
- "Discover people" and striving for "total anonymity" might initially seem incompatible. The idea is to provide for sane defaults, and then allow people to decide how much information they want to disclose. This is the essence of privacy.
- Users often want *some* form of moderation to get a good user experience, which can be seen as a form of censorship. The idea to raise the bar on the basics, the fundamental infrastructure. If individuals or specific communities want certain moderation mechanisms, that is still a compatible requirement.
- Users often want _some_ form of moderation to get a good user experience, which can be seen as a form of censorship. The idea to raise the bar on the basics, the fundamental infrastructure. If individuals or specific communities want certain moderation mechanisms, that is still a compatible requirement.
### Counterpoint 1
@ -115,9 +116,10 @@ Starting from the most basic: what is an account and how do you get one? With mo
Now, we could argue these app providers can do what they want. And they are right, in a very narrow sense. As apps like WeChat (and Google) become general-purpose platforms, they become more and more ingrained in our everyday lives. They start to provide utilities that we absolutely require to work to go about our day, such as paying for food or transportation. This means we need higher standard than this.
Justifications for requiring phone numbers are usually centered around three claims:
1) Avoiding spam
2) Tying your account to your real name, for various reasons
3) Using as a commonly shared identifier as a social network discovery mechanism
1. Avoiding spam
2. Tying your account to your real name, for various reasons
3. Using as a commonly shared identifier as a social network discovery mechanism
Of course, many services require more than phone numbers. E.g. email, other forms of personal data such as voice recording, linking a bank account, and so on.
@ -246,19 +248,11 @@ Corey, Dean, Jacek.
## References
[^1]: Mandatory SIM card registration laws: https://privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws
[^2]: On WeChat keyword censorship: https://citizenlab.ca/2016/11/wechat-china-censorship-one-app-two-systems/
[^3]: Net Neutrality: https://www.eff.org/issues/net-neutrality
[^4]: ISP centralization: https://ilsr.org/repealing-net-neutrality-puts-177-million-americans-at-risk/
[^5]: Incentives Build Robustness in BitTorrent bittorrent.org/bittorrentecon.pdf
[^6]: The Case for Electronic Cash: https://coincenter.org/files/2019-02/the-case-for-electronic-cash-coin-center.pdf
[^6]: The Case for Electronic Cash: https://coincenter.org/files/2019-02/the-case-for-electronic-cash-coin-center.pdf
[^7]: Money, blockchains, and social scalability: http://unenumerated.blogspot.com/2017/02/money-blockchains-and-social-scalability.html
[^8]: Zcash private transactions (partial paywall): https://www.theblockcrypto.com/genesis/48413/an-analysis-of-zcashs-private-transactions
[^9]: Shielded transactions usage (stats page 404s): https://z.cash/support/faq/

34
_posts/2020-04-27-feasibility-discv5.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "Feasibility Study: Discv5"
title: "Feasibility Study: Discv5"
date: 2020-04-27 12:00:00 +0800
name: "Feasibility Study: Discv5"
title: "Feasibility Study: Discv5"
date: 2020-04-27 12:00:00 +0800
author: dean
published: true
permalink: /feasibility-discv5
categories: research
category: research
summary: Looking at discv5 and the theoretical numbers behind finding peers.
discuss: https://discuss.status.im/t/discv5-feasibility-study/1632
---
@ -23,13 +23,13 @@ Namely, we want to ensure censorship resistance to state-level adversaries. One
Considering we are talking about resource restricted devices, let's look at the implications and what we need to consider:
- **Battery consumption** - constant connections like websockets consume a lot of battery life.
- **CPU usage** - certain discovery methods may be CPU incentive, slowing an app down and making it unusable.
- **Bandwidth consumption** - a lot of users will be using data plans, the discovery method needs to be efficient in order to accommodate those users without using up significant portions of their data plans.
- **Short connection windows** - the discovery algorithm needs to be low latency, that means it needs to return results fast. This is because many users will only have the app open for a short amount of time.
- **Not publicly connectable** - There is a good chance that most resource restricted devices are not publicly connectable.
- **Battery consumption** - constant connections like websockets consume a lot of battery life.
- **CPU usage** - certain discovery methods may be CPU incentive, slowing an app down and making it unusable.
- **Bandwidth consumption** - a lot of users will be using data plans, the discovery method needs to be efficient in order to accommodate those users without using up significant portions of their data plans.
- **Short connection windows** - the discovery algorithm needs to be low latency, that means it needs to return results fast. This is because many users will only have the app open for a short amount of time.
- **Not publicly connectable** - There is a good chance that most resource restricted devices are not publicly connectable.
For a node to be able to participate as both a provider, and a consumer in the discovery method. Meaning a node both reads from other nodes' stored DHTs and hosts the DHT for other nodes to read from, it needs to be publically connectable. This means another node must be able to connect to some public IP of the given node.
For a node to be able to participate as both a provider, and a consumer in the discovery method. Meaning a node both reads from other nodes' stored DHTs and hosts the DHT for other nodes to read from, it needs to be publically connectable. This means another node must be able to connect to some public IP of the given node.
With devices that are behind a NAT, this is easier said than done. Especially mobile devices, that when connected to 4G LTE networks are often stuck behind a symmetric NAT, drastically reducing the the succeess rate of NAT traversal. Keeping this in mind, it becomes obvious that most resource restricted devices will be consumers rather than providers due to this technical limitation.
@ -38,8 +38,9 @@ In order to answer our questions, we formulated the problem with a simple method
## Overview
The main things we wanted to investigate was the overhead on finding a peer. This means we wanted to look at both the bandwidth, latency and effectiveness of this. There are 2 methods which we can use to find a peer:
- We can find a peer with a specific ID, using normal lookup methods as documented by Kademlia.
- We can find a peer that advertises a capability, this is possible using either capabilities advertised in the ENR or through [topic tables](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement).
- We can find a peer with a specific ID, using normal lookup methods as documented by Kademlia.
- We can find a peer that advertises a capability, this is possible using either capabilities advertised in the ENR or through [topic tables](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement).
## Feasbility
@ -76,7 +77,8 @@ Research has shown that finding a node in the DHT has a relatively low effect on
One of the main blockers right now is figuring out what the CPU and memory usage of discv5 is on mobile phones, this is a large blocker as it affects one of the core problems for us. We need to consider whether discv5 is an upgrade as it allows us to simplify our current discovery process or if it is too much of an overhead for resource restricted devices. The topic table feature could largely enhance discovery however it is not yet implemented. Given that CPU and memory isn't too high, discv5 could probably be used as the other issues are more "features" than large scale issues. Implementing it would already reduce the ability for state level adversaries to censor our nodes.
## Acknowledgements
- Oskar Thoren
- Dmitry Shmatko
- Kim De Mey
- Corey Petty
- Oskar Thoren
- Dmitry Shmatko
- Kim De Mey
- Corey Petty

23
_posts/2020-04-9-kademlia-to-discv5.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "From Kademlia to Discv5"
title: "From Kademlia to Discv5"
date: 2020-04-9 16:00:00 +0100
name: "From Kademlia to Discv5"
title: "From Kademlia to Discv5"
date: 2020-04-9 16:00:00 +0100
author: dean
published: true
permalink: /kademlia-to-discv5
categories: research
category: research
summary: A quick history of discovery in peer-to-peer networks, along with a look into discv4 and discv5, detailing what they are, how they work and where they differ.
---
@ -48,9 +48,10 @@ XOR 10011001
The top in decimal numbers means that the distance between `153` and `50` is `171`.
There are several reasons why `XOR` was taken:
1. The distance from one ID to itself will be `0`.
2. Distance is symmetric, A to B is the same as B to A.
3. Follows triangle inequality, if `A`, `B` and `C` are points on a triangle then the distance `A` to `B` is closer or equal to that of `A` to `C` plus the one from `B` to `C`.
1. The distance from one ID to itself will be `0`.
2. Distance is symmetric, A to B is the same as B to A.
3. Follows triangle inequality, if `A`, `B` and `C` are points on a triangle then the distance `A` to `B` is closer or equal to that of `A` to `C` plus the one from `B` to `C`.
In summary, this distance function allows a node to decide what is "close" to it and make decisions based on that "closeness".
@ -67,11 +68,11 @@ For Kademlia nodes to support these functions, there are several messages with w
- `FINDNODE` - Returns the closest nodes requested to a given ID.
- `FINDVALUE` - The same as `FINDNODE`, except if a node stores the specific value it will return it directly.
*This is a **very** simplified explanation of Kademlia and skips various important details. For the full description, make sure to check out the [paper](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf) or a more in-depth [design specification](http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html)*
_This is a **very** simplified explanation of Kademlia and skips various important details. For the full description, make sure to check out the [paper](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf) or a more in-depth [design specification](http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html)_
## Discv4
Now after that history lesson, we finally get to discv4 (which stands for discovery v4), Ethereum's current node discovery protocol. The protocol itself is essentially based off of Kademlia, however it does away with certain aspects of it. For example, it does away with any usage of the value part of the DHT.
Now after that history lesson, we finally get to discv4 (which stands for discovery v4), Ethereum's current node discovery protocol. The protocol itself is essentially based off of Kademlia, however it does away with certain aspects of it. For example, it does away with any usage of the value part of the DHT.
Kademlia is mainly used for the organisation of the network, so we only use the routing table to locate other nodes. Due to the fact that discv4 doesn't use the value portion of the DHT at all, we can throw away the `FINDVALUE` and `STORE` commands described by Kademlia.
@ -81,7 +82,7 @@ Additionally, discv4 adds mutual endpoint verification. This is meant to ensure
Finally, all discv4 nodes are expected to maintain up-to-date ENR records. These contain information about a node. They can be requested from any node using a discv4-specific packet called `ENRRequest`.
*If you want some more details on ENRs, check out one of my posts ["Network Addresses in Ethereum"](https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/)*
_If you want some more details on ENRs, check out one of my posts ["Network Addresses in Ethereum"](https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/)_
Discv4 comes with its own range of problems however. Let's look at a few of them.
@ -103,7 +104,7 @@ Logarithmic distance means we first calculate the distance and then run it throu
log2(A xor B)
```
And the second, more important change, is that discv5 aims at solving one of the biggest issues of discv4: the differentiation of sub-protocols. It does this by adding topic tables. Topic tables are [first in first out](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) lists that contain nodes which have advertised that they provide a specific service. Nodes get themselves added to this list by registering `ads` on their peers.
And the second, more important change, is that discv5 aims at solving one of the biggest issues of discv4: the differentiation of sub-protocols. It does this by adding topic tables. Topic tables are [first in first out](<https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)>) lists that contain nodes which have advertised that they provide a specific service. Nodes get themselves added to this list by registering `ads` on their peers.
As of writing, there is still an issue with this proposal. There is currently no efficient way for a node to place `ads` on multiple peers, since it would require separate requests for every peer which is inefficient in a large-scale network.

178
_posts/2020-07-01-problem-4.md Normal file
View File

@ -0,0 +1,178 @@
---
layout: post
name: "Problem 4 Wallet stacking sats block height"
title: "Problem 4 Wallet stacking sats block height"
date: 2020-07-01 12:00:00 +0800
author: franck
published: true
permalink: /status-ipsum-4
category: problem
summary: "Status ipsum lorem exequoe"
image: /assets/img/js-waku-gist.png
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
---
For the past 3 months, we have been working on bringing Waku v2 to the browser.
Our aim is to empower dApps with Waku v2, and it led to the creation of a new library.
We believe now is good time to introduce it!
## Waku v2
First, let's review what Waku v2 is and what problem it is trying to solve.
Waku v2 comes from a need to have a more scalable, better optimised solution for the Status app to achieve decentralised
communications on resource restricted devices (i.e., mobile phones).
The Status chat feature was initially built over Whisper.
However, Whisper has a number of caveats which makes it inefficient for mobile phones.
For example, with Whisper, all devices are receiving all messages which is not ideal for limited data plans.
To remediate this, a Waku mode (then Waku v1), based on devp2p, was introduced.
To further enable web and restricted resource environments, Waku v2 was created based on libp2p.
The migration of the Status chat feature to Waku v2 is currently in progress.
We see the need of such solution in the broader Ethereum ecosystem, beyond Status.
This is why we are building Waku v2 as a decentralised communication platform for all to use and build on.
If you want to read more about Waku v2 and what it aims to achieve,
checkout [What's the Plan for Waku v2?](/waku-v2-plan).
Since last year, we have been busy defining and implementing Waku v2 protocols in [nim-waku](https://github.com/status-im/nim-waku),
from which you can build [wakunode2](https://github.com/status-im/nim-waku#wakunode).
Wakunode2 is an adaptive and modular Waku v2 node,
it allows users to run their own node and use the Waku v2 protocols they need.
The nim-waku project doubles as a library, that can be used to add Waku v2 support to native applications.
## Waku v2 in the browser
We believe that dApps and wallets can benefit from the Waku network in several ways.
For some dApps, it makes sense to enable peer-to-peer communications.
For others, machine-to-machine communications would be a great asset.
For example, in the case of a DAO,
Waku could be used for gas-less voting.
Enabling the DAO to notify their users of a new vote,
and users to vote without interacting with the blockchain and spending gas.
[Murmur](https://github.com/status-im/murmur) was the first attempt to bring Whisper to the browser,
acting as a bridge between devp2p and libp2p.
Once Waku v2 was started and there was a native implementation on top of libp2p,
a [chat POC](https://github.com/vacp2p/waku-web-chat) was created to demonstrate the potential of Waku v2
in web environment.
It showed how using js-libp2p with few modifications enabled access to the Waku v2 network.
There was still some unresolved challenges.
For example, nim-waku only support TCP connections which are not supported by browser applications.
Hence, to connect to other node, the POC was connecting to a NodeJS proxy application using websockets,
which in turn could connect to wakunode2 via TCP.
However, to enable dApp and Wallet developers to easily integrate Waku in their product,
we need to give them a library that is easy to use and works out of the box:
introducing [JS-Waku](https://github.com/status-im/js-waku).
JS-Waku is a JavaScript library that allows your dApp, wallet or other web app to interact with the Waku v2 network.
It is available right now on [npm](https://www.npmjs.com/package/js-waku):
`npm install js-waku`.
As it is written in TypeScript, types are included in the npm package to allow easy integration with TypeScript, ClojureScript and other typed languages that compile to JavaScript.
Key Waku v2 protocols are already available:
[message](https://rfc.vac.dev/spec/14/), [store](https://rfc.vac.dev/spec/13/), [relay](https://rfc.vac.dev/spec/11/) and [light push](https://rfc.vac.dev/spec/19/),
enabling your dApp to:
- Send and receive near-instant messages on the Waku network (relay),
- Query nodes for messages that may have been missed, e.g. due to poor cellular network (store),
- Send messages with confirmations (light push).
JS-Waku needs to operate in the same context from which Waku v2 was born:
a restricted environment were connectivity or uptime are not guaranteed;
JS-Waku brings Waku v2 to the browser.
## Achievements so far
We focused the past month on developing a [ReactJS Chat App](https://status-im.github.io/js-waku/).
The aim was to create enough building blocks in JS-Waku to enable this showcase web app that
we now [use for dogfooding](https://github.com/status-im/nim-waku/issues/399) purposes.
Most of the effort was on getting familiar with the [js-libp2p](https://github.com/libp2p/js-libp2p) library
that we heavily rely on.
JS-Waku is the second implementation of Waku v2 protocol,
so a lot of effort on interoperability was needed.
For example, to ensure compatibility with the nim-waku reference implementation,
we run our [tests against wakunode2](https://github.com/status-im/js-waku/blob/90c90dea11dfd1277f530cf5d683fb92992fe141/src/lib/waku_relay/index.spec.ts#L137) as part of the CI.
This interoperability effort helped solidify the current Waku v2 specifications:
By clarifying the usage of topics
([#327](https://github.com/vacp2p/rfc/issues/327), [#383](https://github.com/vacp2p/rfc/pull/383)),
fix discrepancies between specs and nim-waku
([#418](https://github.com/status-im/nim-waku/issues/418), [#419](https://github.com/status-im/nim-waku/issues/419))
and fix small nim-waku & nim-libp2p bugs
([#411](https://github.com/status-im/nim-waku/issues/411), [#439](https://github.com/status-im/nim-waku/issues/439)).
To fully access the waku network, JS-Waku needs to enable web apps to connect to nim-waku nodes.
A standard way to do so is using secure websockets as it is not possible to connect directly to a TCP port from the browser.
Unfortunately websocket support is not yet available in [nim-libp2p](https://github.com/status-im/nim-libp2p/issues/407) so
we ended up deploying [websockify](https://github.com/novnc/websockify) alongside wakunode2 instances.
As we built the [web chat app](https://github.com/status-im/js-waku/tree/main/examples/web-chat),
we were able to fine tune the API to provide a simple and succinct interface.
You can start a node, connect to other nodes and send a message in less than ten lines of code:
```javascript
import { Waku } from "js-waku";
const waku = await Waku.create({});
const nodes = await getStatusFleetNodes();
await Promise.all(nodes.map((addr) => waku.dial(addr)));
const msg = WakuMessage.fromUtf8String(
"Here is a message!",
"/my-cool-app/1/my-use-case/proto"
);
await waku.relay.send(msg);
```
We have also put a bounty at [0xHack](https://0xhack.dev/) for using JS-Waku
and running a [workshop](https://vimeo.com/551509621).
We were thrilled to have a couple of hackers create new software using our libraries.
One of the projects aimed to create a decentralised, end-to-end encrypted messenger app,
similar to what the [ETH-DM](https://rfc.vac.dev/spec/20/) protocol aims to achieve.
Another project was a decentralised Twitter platform.
Such projects allow us to prioritize the work on JS-Waku and understand how DevEx can be improved.
As more developers use JS-Waku, we will evolve the API to allow for more custom and fine-tune usage of the network
while preserving this out of the box experience.
## What's next?
Next, we are directing our attention towards [Developer Experience](https://github.com/status-im/js-waku/issues/68).
We already have [documentation](https://www.npmjs.com/package/js-waku) available but we want to provide more:
[Tutorials](https://github.com/status-im/js-waku/issues/56), various examples
and showing how [JS-Waku can be used with Web3](https://github.com/status-im/js-waku/issues/72).
By prioritizing DevEx we aim to enable JS-Waku integration in dApps and wallets.
We think JS-Waku builds a strong case for machine-to-machine (M2M) communications.
The first use cases we are looking into are dApp notifications:
Enabling dApp to notify their user directly in their wallets!
Leveraging Waku as a decentralised infrastructure and standard so that users do not have to open their dApp to be notified
of events such as DAO voting.
We already have some POC in the pipeline to enable voting and polling on the Waku network,
allowing users to save gas by **not** broadcasting each individual vote on the blockchain.
To facilitate said applications, we are looking at improving integration with Web3 providers by providing examples
of signing, validating, encrypting and decrypting messages using Web3.
Waku is privacy conscious, so we will also provide signature and encryption examples decoupled from users' Ethereum identity.
As you can read, we have grand plans for JS-Waku and Waku v2.
There is a lot to do, and we would love some help so feel free to
check out the new role in our team:
[js-waku: Wallet & Dapp Integration Developer](https://status.im/our_team/jobs.html?gh_jid=3157894).
We also have a number of [positions](https://status.im/our_team/jobs.html) open to work on Waku protocol and nim-waku.
If you are as excited as us by JS-Waku, why not build a dApp with it?
You can find documentation on the [npmjs page](https://www.npmjs.com/package/js-waku).
Whether you are a developer, you can come chat with us using [WakuJS Web Chat](https://status-im.github.io/js-waku/)
or [chat2](https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md).
You can get support on Discord [#waku-support (dev support)](https://discord.gg/VChNsDdj).
If you have any ideas on how Waku could enable a specific dapp or use case, do share, we are always keen to hear it.

46
_posts/2020-07-01-waku-v2-pitch.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "What's the Plan for Waku v2?"
title: "What's the Plan for Waku v2?"
date: 2020-07-01 12:00:00 +0800
name: "What's the Plan for Waku v2?"
title: "What's the Plan for Waku v2?"
date: 2020-07-01 12:00:00 +0800
author: oskarth
published: true
permalink: /waku-v2-plan
categories: research
category: research
summary: Read about our plans for Waku v2, moving to libp2p, better routing, adaptive nodes and accounting!
image: /assets/img/status_scaling_model_fig4.png
discuss: https://forum.vac.dev/t/waku-version-2-pitch/52
@ -14,7 +14,7 @@ discuss: https://forum.vac.dev/t/waku-version-2-pitch/52
**tldr: The Waku network is fragile and doesn't scale. Here's how to solve it.**
*NOTE: This post was originally written with Status as a primary use case in mind, which reflects how we talk about some problems here. However, Waku v2 is a general-purpose private p2p messaging protocol, especially for people running in resource restricted environments.*
_NOTE: This post was originally written with Status as a primary use case in mind, which reflects how we talk about some problems here. However, Waku v2 is a general-purpose private p2p messaging protocol, especially for people running in resource restricted environments._
# Problem
@ -26,12 +26,12 @@ Based on user acquisition models, the initial goal is to support 100k DAU in Sep
With the Status Scaling Model we have studied the current bottlenecks as a function of concurrent users (CCU) and daily active users (DAU). Here are the conclusions.
****1. Connection limits****. With 100 full nodes we reach ~10k CCU based on connection limits. This can primarily be addressed by increasing the number of nodes (cluster or user operated). This assumes node discovery works. It is also worth investigating the limitations of max number of connections, though this is likely to be less relevant for user-operated nodes. For a user-operated network, this means 1% of users have to run a full node. See Fig 1-2.
\***\*1. Connection limits\*\***. With 100 full nodes we reach ~10k CCU based on connection limits. This can primarily be addressed by increasing the number of nodes (cluster or user operated). This assumes node discovery works. It is also worth investigating the limitations of max number of connections, though this is likely to be less relevant for user-operated nodes. For a user-operated network, this means 1% of users have to run a full node. See Fig 1-2.
****2. Bandwidth as a bottleneck****. We notice that memory usage appears to not be
\***\*2. Bandwidth as a bottleneck\*\***. We notice that memory usage appears to not be
the primary bottleneck for full nodes, and the bottleneck is still bandwidth. To support 10k DAU, and full nodes with an amplification factor of 25 the required Internet speed is ~50 Mbps, which is a fast home Internet connection. For ~100k DAU only cloud-operated nodes can keep up (500 Mbps). See Fig 3-5.
****3. Amplification factors****. Reducing amplification factors with better routing, would have a high impact, but it is likely we'd need additional measures as well, such as topic sharding or similar. See Fig 8-13.
\***\*3. Amplification factors\*\***. Reducing amplification factors with better routing, would have a high impact, but it is likely we'd need additional measures as well, such as topic sharding or similar. See Fig 8-13.
Figure 1-5:
@ -50,12 +50,12 @@ What we need to do is:
Doing this means the Waku network will be able to scale, and doing so in the right way, in a robust fashion. What would a fragile way of scaling be? Increasing our reliance on a Status Pte Ltd operated cluster which would paint us in a corner where we:
- keep increasing requirements for Internet speed for full nodes
- are vulnerable to censorship and attacks
- have to control the topology in an artifical manner to keep up with load
- basically re-invent a traditional centralized client-server app with extra steps
- deliberately ignore most of our principles
- risk the network being shut down when we run out of cash
- keep increasing requirements for Internet speed for full nodes
- are vulnerable to censorship and attacks
- have to control the topology in an artifical manner to keep up with load
- basically re-invent a traditional centralized client-server app with extra steps
- deliberately ignore most of our principles
- risk the network being shut down when we run out of cash
# Appetite
@ -79,7 +79,6 @@ Let's first look at the baseline, and then go into some of the tracks and their
Here's where we are at now. In reality, the amplification factor are likely even worse than this (15 in the graph below), up to 20-30. Especially with an open network, where we can't easily control connectivity and availability of nodes. Left unchecked, with a full mesh, it could even go as high x100, though this is likely excessive and can be dialed down. See scaling model for more details.
![](assets/img/waku_v1_routing_small.png)
## Track 1 - Move to libp2p
@ -100,7 +99,6 @@ Why can't we use Waku topics for routing directly? PubSub over libp2p isn't buil
![](assets/img/waku_v2_routing_flood_small.png)
Moving to FloodSub over libp2p would also be an opportunity to clean up and simplify some components that are no longer needed in the Waku v1 protocol, see point below.
Very experimental and incomplete libp2p support can be found in the nim-waku repo under v2: <https://github.com/status-im/nim-waku>
@ -201,9 +199,9 @@ The general shape of the solution is inspired by the Swarm model, where we do ac
While accounting for individual resource usage is useful, for the ultimate end user experience we can ideally account for other things such as:
- end to end delivery
- online time
- completeness of storage
- end to end delivery
- online time
- completeness of storage
This can be gradually enhanced and strengthened, for example with proofs, consistency checks, Quality of Service, reputation systems. See <https://discuss.status.im/t/network-incentivisation-first-draft/1037> for one attempt to provide stronger guarantees with periodic consistency checks and a shared fund mechanism. And <https://forum.vac.dev/t/incentivized-messaging-using-validity-proofs/51> for using validity proofs and removing liveness requirement for settlement.
@ -227,14 +225,14 @@ In accounting phase it is largely assumed nodes are honest. What happens when th
For light nodes:
- if they don't, they get disconnected
- prepayment (especially to "high value" nodes)
- if they don't, they get disconnected
- prepayment (especially to "high value" nodes)
For full nodes:
- multiple nodes reporting to agree, where truth becomes a shelling point
- use eigentrust
- staking for discovery visibility with slashing
- multiple nodes reporting to agree, where truth becomes a shelling point
- use eigentrust
- staking for discovery visibility with slashing
### 5. Settlement PoC

178
_posts/2020-09-28-problem-2.md Normal file
View File

@ -0,0 +1,178 @@
---
layout: post
name: "Problem No. 2, very long title Bitcoin ipsum dolor sit amet. Peer-to-peer whitepaper SHA-256 bitcoin."
title: "Problem No. 2, very long title Bitcoin ipsum dolor sit amet. Peer-to-peer whitepaper SHA-256 bitcoin."
date: 2020-09-28 12:00:00 +0800
author: franck
published: true
permalink: /status-ipsum-2
category: problem
summary: "Status ipsum lorem exequoe"
image: /assets/img/js-waku-gist.png
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
---
For the past 3 months, we have been working on bringing Waku v2 to the browser.
Our aim is to empower dApps with Waku v2, and it led to the creation of a new library.
We believe now is good time to introduce it!
## Waku v2
First, let's review what Waku v2 is and what problem it is trying to solve.
Waku v2 comes from a need to have a more scalable, better optimised solution for the Status app to achieve decentralised
communications on resource restricted devices (i.e., mobile phones).
The Status chat feature was initially built over Whisper.
However, Whisper has a number of caveats which makes it inefficient for mobile phones.
For example, with Whisper, all devices are receiving all messages which is not ideal for limited data plans.
To remediate this, a Waku mode (then Waku v1), based on devp2p, was introduced.
To further enable web and restricted resource environments, Waku v2 was created based on libp2p.
The migration of the Status chat feature to Waku v2 is currently in progress.
We see the need of such solution in the broader Ethereum ecosystem, beyond Status.
This is why we are building Waku v2 as a decentralised communication platform for all to use and build on.
If you want to read more about Waku v2 and what it aims to achieve,
checkout [What's the Plan for Waku v2?](/waku-v2-plan).
Since last year, we have been busy defining and implementing Waku v2 protocols in [nim-waku](https://github.com/status-im/nim-waku),
from which you can build [wakunode2](https://github.com/status-im/nim-waku#wakunode).
Wakunode2 is an adaptive and modular Waku v2 node,
it allows users to run their own node and use the Waku v2 protocols they need.
The nim-waku project doubles as a library, that can be used to add Waku v2 support to native applications.
## Waku v2 in the browser
We believe that dApps and wallets can benefit from the Waku network in several ways.
For some dApps, it makes sense to enable peer-to-peer communications.
For others, machine-to-machine communications would be a great asset.
For example, in the case of a DAO,
Waku could be used for gas-less voting.
Enabling the DAO to notify their users of a new vote,
and users to vote without interacting with the blockchain and spending gas.
[Murmur](https://github.com/status-im/murmur) was the first attempt to bring Whisper to the browser,
acting as a bridge between devp2p and libp2p.
Once Waku v2 was started and there was a native implementation on top of libp2p,
a [chat POC](https://github.com/vacp2p/waku-web-chat) was created to demonstrate the potential of Waku v2
in web environment.
It showed how using js-libp2p with few modifications enabled access to the Waku v2 network.
There was still some unresolved challenges.
For example, nim-waku only support TCP connections which are not supported by browser applications.
Hence, to connect to other node, the POC was connecting to a NodeJS proxy application using websockets,
which in turn could connect to wakunode2 via TCP.
However, to enable dApp and Wallet developers to easily integrate Waku in their product,
we need to give them a library that is easy to use and works out of the box:
introducing [JS-Waku](https://github.com/status-im/js-waku).
JS-Waku is a JavaScript library that allows your dApp, wallet or other web app to interact with the Waku v2 network.
It is available right now on [npm](https://www.npmjs.com/package/js-waku):
`npm install js-waku`.
As it is written in TypeScript, types are included in the npm package to allow easy integration with TypeScript, ClojureScript and other typed languages that compile to JavaScript.
Key Waku v2 protocols are already available:
[message](https://rfc.vac.dev/spec/14/), [store](https://rfc.vac.dev/spec/13/), [relay](https://rfc.vac.dev/spec/11/) and [light push](https://rfc.vac.dev/spec/19/),
enabling your dApp to:
- Send and receive near-instant messages on the Waku network (relay),
- Query nodes for messages that may have been missed, e.g. due to poor cellular network (store),
- Send messages with confirmations (light push).
JS-Waku needs to operate in the same context from which Waku v2 was born:
a restricted environment were connectivity or uptime are not guaranteed;
JS-Waku brings Waku v2 to the browser.
## Achievements so far
We focused the past month on developing a [ReactJS Chat App](https://status-im.github.io/js-waku/).
The aim was to create enough building blocks in JS-Waku to enable this showcase web app that
we now [use for dogfooding](https://github.com/status-im/nim-waku/issues/399) purposes.
Most of the effort was on getting familiar with the [js-libp2p](https://github.com/libp2p/js-libp2p) library
that we heavily rely on.
JS-Waku is the second implementation of Waku v2 protocol,
so a lot of effort on interoperability was needed.
For example, to ensure compatibility with the nim-waku reference implementation,
we run our [tests against wakunode2](https://github.com/status-im/js-waku/blob/90c90dea11dfd1277f530cf5d683fb92992fe141/src/lib/waku_relay/index.spec.ts#L137) as part of the CI.
This interoperability effort helped solidify the current Waku v2 specifications:
By clarifying the usage of topics
([#327](https://github.com/vacp2p/rfc/issues/327), [#383](https://github.com/vacp2p/rfc/pull/383)),
fix discrepancies between specs and nim-waku
([#418](https://github.com/status-im/nim-waku/issues/418), [#419](https://github.com/status-im/nim-waku/issues/419))
and fix small nim-waku & nim-libp2p bugs
([#411](https://github.com/status-im/nim-waku/issues/411), [#439](https://github.com/status-im/nim-waku/issues/439)).
To fully access the waku network, JS-Waku needs to enable web apps to connect to nim-waku nodes.
A standard way to do so is using secure websockets as it is not possible to connect directly to a TCP port from the browser.
Unfortunately websocket support is not yet available in [nim-libp2p](https://github.com/status-im/nim-libp2p/issues/407) so
we ended up deploying [websockify](https://github.com/novnc/websockify) alongside wakunode2 instances.
As we built the [web chat app](https://github.com/status-im/js-waku/tree/main/examples/web-chat),
we were able to fine tune the API to provide a simple and succinct interface.
You can start a node, connect to other nodes and send a message in less than ten lines of code:
```javascript
import { Waku } from "js-waku";
const waku = await Waku.create({});
const nodes = await getStatusFleetNodes();
await Promise.all(nodes.map((addr) => waku.dial(addr)));
const msg = WakuMessage.fromUtf8String(
"Here is a message!",
"/my-cool-app/1/my-use-case/proto"
);
await waku.relay.send(msg);
```
We have also put a bounty at [0xHack](https://0xhack.dev/) for using JS-Waku
and running a [workshop](https://vimeo.com/551509621).
We were thrilled to have a couple of hackers create new software using our libraries.
One of the projects aimed to create a decentralised, end-to-end encrypted messenger app,
similar to what the [ETH-DM](https://rfc.vac.dev/spec/20/) protocol aims to achieve.
Another project was a decentralised Twitter platform.
Such projects allow us to prioritize the work on JS-Waku and understand how DevEx can be improved.
As more developers use JS-Waku, we will evolve the API to allow for more custom and fine-tune usage of the network
while preserving this out of the box experience.
## What's next?
Next, we are directing our attention towards [Developer Experience](https://github.com/status-im/js-waku/issues/68).
We already have [documentation](https://www.npmjs.com/package/js-waku) available but we want to provide more:
[Tutorials](https://github.com/status-im/js-waku/issues/56), various examples
and showing how [JS-Waku can be used with Web3](https://github.com/status-im/js-waku/issues/72).
By prioritizing DevEx we aim to enable JS-Waku integration in dApps and wallets.
We think JS-Waku builds a strong case for machine-to-machine (M2M) communications.
The first use cases we are looking into are dApp notifications:
Enabling dApp to notify their user directly in their wallets!
Leveraging Waku as a decentralised infrastructure and standard so that users do not have to open their dApp to be notified
of events such as DAO voting.
We already have some POC in the pipeline to enable voting and polling on the Waku network,
allowing users to save gas by **not** broadcasting each individual vote on the blockchain.
To facilitate said applications, we are looking at improving integration with Web3 providers by providing examples
of signing, validating, encrypting and decrypting messages using Web3.
Waku is privacy conscious, so we will also provide signature and encryption examples decoupled from users' Ethereum identity.
As you can read, we have grand plans for JS-Waku and Waku v2.
There is a lot to do, and we would love some help so feel free to
check out the new role in our team:
[js-waku: Wallet & Dapp Integration Developer](https://status.im/our_team/jobs.html?gh_jid=3157894).
We also have a number of [positions](https://status.im/our_team/jobs.html) open to work on Waku protocol and nim-waku.
If you are as excited as us by JS-Waku, why not build a dApp with it?
You can find documentation on the [npmjs page](https://www.npmjs.com/package/js-waku).
Whether you are a developer, you can come chat with us using [WakuJS Web Chat](https://status-im.github.io/js-waku/)
or [chat2](https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md).
You can get support on Discord [#waku-support (dev support)](https://discord.gg/VChNsDdj).
If you have any ideas on how Waku could enable a specific dapp or use case, do share, we are always keen to hear it.

32
_posts/2020-09-28-waku-v2-update.md
View File

@ -1,12 +1,12 @@
---
layout: post
name: "Waku v2 Update"
title: "Waku v2 Update"
date: 2020-09-28 12:00:00 +0800
name: "Waku v2 Update"
title: "Waku v2 Update"
date: 2020-09-28 12:00:00 +0800
author: oskarth
published: true
permalink: /waku-v2-update
categories: research
category: research
summary: A research log. Read on to find out what is going on with Waku v2, a messaging protocol. What has been happening? What is coming up next?
image: /assets/img/vac.png
discuss: https://forum.vac.dev/t/discussion-waku-v2-update/56
@ -72,16 +72,16 @@ Now that we know what the current state is, what is still missing? what are the
While we are getting closer to closing out work for track 1, there are still a few things missing from the initial scope:
1) Store and filter protocols need to be finished. This means basic spec, implementation, API integration and proven to work in a testnet. All of these are work in progress and expected to be done very soon. Once the store protocol is done in a basic form, it needs further improvements to make it production ready, at least on a spec/basic implementation level.
1. Store and filter protocols need to be finished. This means basic spec, implementation, API integration and proven to work in a testnet. All of these are work in progress and expected to be done very soon. Once the store protocol is done in a basic form, it needs further improvements to make it production ready, at least on a spec/basic implementation level.
2) Core integration was mentioned in scope for track 1 initially. This work has stalled a bit, largely due to organizational bandwidth and priorities. While there is a Nim Node API that in theory is ready to be used, having it be used in e.g. Status desktop or mobile app is a different matter. The team responsible for this at Status ([status-nim](github.com/status-im/status-nim)) has been making progress on getting nim-waku v1 integrated, and is expected to look into nim-waku v2 integration soon. One thing that makes this a especially tricky is the difference in interface between Waku v1 and v2, which brings
us too...
2. Core integration was mentioned in scope for track 1 initially. This work has stalled a bit, largely due to organizational bandwidth and priorities. While there is a Nim Node API that in theory is ready to be used, having it be used in e.g. Status desktop or mobile app is a different matter. The team responsible for this at Status ([status-nim](github.com/status-im/status-nim)) has been making progress on getting nim-waku v1 integrated, and is expected to look into nim-waku v2 integration soon. One thing that makes this a especially tricky is the difference in interface between Waku v1 and v2, which brings
us too...
3) Companion spec for encryption. As part of simplifying the protocol, the routing is decoupled from the encryption in v2 ([1](https://github.com/vacp2p/specs/issues/158), [2](https://github.com/vacp2p/specs/issues/181)). There are multiple layers of encryption at play here, and we need to figure out a design that makes sense for various use cases (dapps using Waku on their own, Status app, etc).
3. Companion spec for encryption. As part of simplifying the protocol, the routing is decoupled from the encryption in v2 ([1](https://github.com/vacp2p/specs/issues/158), [2](https://github.com/vacp2p/specs/issues/181)). There are multiple layers of encryption at play here, and we need to figure out a design that makes sense for various use cases (dapps using Waku on their own, Status app, etc).
4) Bridge implementation. The spec is done and we know how it should work, but it needs to be implemented.
4. Bridge implementation. The spec is done and we know how it should work, but it needs to be implemented.
5) General tightening up of specs and implementation.
5. General tightening up of specs and implementation.
While this might seem like a lot, a lot has been done already, and the majority of the remaining tasks are more amendable to be pursued in parallel with other efforts. It is also worth mentioning that part of track 2 and 3 have been started, in the form of moving to GossipSub (amplification factors) and basics of adaptive nodes (multiple protocols). This is in addition to things like Waku Web which were not part of the initial scope.
@ -89,16 +89,16 @@ While this might seem like a lot, a lot has been done already, and the majority
Aside from the things mentioned above, what is coming up next? There are a few areas of interest, mentioned in no particular order. For track 2 and 3, see previous post for more details.
1) Better routing (track 2). While we are already building on top of GossipSub, we still need to explore things like topic sharding in more detail to further reduce amplification factors.
1. Better routing (track 2). While we are already building on top of GossipSub, we still need to explore things like topic sharding in more detail to further reduce amplification factors.
2) Accounting and user-run nodes (track 3). With store and filter protocol getting ready, we can start to implement accounting and light connection game for incentivization in a bottom up and iterative manner.
2. Accounting and user-run nodes (track 3). With store and filter protocol getting ready, we can start to implement accounting and light connection game for incentivization in a bottom up and iterative manner.
3) Privacy research. Study better and more rigorous privacy guarantees. E.g. how FloodSub/GossipSub behaves for common threat models, and how custom packet
format can improve things like unlinkability.
3. Privacy research. Study better and more rigorous privacy guarantees. E.g. how FloodSub/GossipSub behaves for common threat models, and how custom packet
format can improve things like unlinkability.
4) zkSnarks RLN for spam protection and incentivization. We studied this [last year](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks) and recent developments have made this relevant to study again. Create an [experimental spec/PoC](https://github.com/vacp2p/specs/issues/189) as an extension to the relay protocol. Kudos to Barry Whitehat and others like Kobi Gurkan and Koh Wei Jie for pushing this!
4. zkSnarks RLN for spam protection and incentivization. We studied this [last year](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks) and recent developments have made this relevant to study again. Create an [experimental spec/PoC](https://github.com/vacp2p/specs/issues/189) as an extension to the relay protocol. Kudos to Barry Whitehat and others like Kobi Gurkan and Koh Wei Jie for pushing this!
5) Ethereum M2M messaging. Being able to run in the browser opens up a lot of doors, and there is an opportunity here to enable things like a decentralized WalletConnect, multi-sig transactions, voting and similar use cases. This was the original goal of Whisper, and we'd like to deliver on that.
5. Ethereum M2M messaging. Being able to run in the browser opens up a lot of doors, and there is an opportunity here to enable things like a decentralized WalletConnect, multi-sig transactions, voting and similar use cases. This was the original goal of Whisper, and we'd like to deliver on that.
As you can tell, quite a lot of thing! Luckily, we have two people joining as protocol engineers soon, which will bring much needed support for the current team of ~2-2.5 people. More details to come in further updates.

178
_posts/2020-11-10-problem-3.md Normal file
View File

@ -0,0 +1,178 @@
---
layout: post
name: "Problem No. 3 Inputs cryptocurrency key pair timestamp server, difficulty outputs! Key pair, halvening consensus "
title: "Problem No. 3 Inputs cryptocurrency key pair timestamp server, difficulty outputs! Key pair, halvening consensus "
date: 2020-11-10 12:00:00 +0800
author: franck
published: true
permalink: /status-ipsum-3
category: problem
summary: "Status ipsum lorem exequoe"
image: /assets/img/js-waku-gist.png
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
---
For the past 3 months, we have been working on bringing Waku v2 to the browser.
Our aim is to empower dApps with Waku v2, and it led to the creation of a new library.
We believe now is good time to introduce it!
## Waku v2
First, let's review what Waku v2 is and what problem it is trying to solve.
Waku v2 comes from a need to have a more scalable, better optimised solution for the Status app to achieve decentralised
communications on resource restricted devices (i.e., mobile phones).
The Status chat feature was initially built over Whisper.
However, Whisper has a number of caveats which makes it inefficient for mobile phones.
For example, with Whisper, all devices are receiving all messages which is not ideal for limited data plans.
To remediate this, a Waku mode (then Waku v1), based on devp2p, was introduced.
To further enable web and restricted resource environments, Waku v2 was created based on libp2p.
The migration of the Status chat feature to Waku v2 is currently in progress.
We see the need of such solution in the broader Ethereum ecosystem, beyond Status.
This is why we are building Waku v2 as a decentralised communication platform for all to use and build on.
If you want to read more about Waku v2 and what it aims to achieve,
checkout [What's the Plan for Waku v2?](/waku-v2-plan).
Since last year, we have been busy defining and implementing Waku v2 protocols in [nim-waku](https://github.com/status-im/nim-waku),
from which you can build [wakunode2](https://github.com/status-im/nim-waku#wakunode).
Wakunode2 is an adaptive and modular Waku v2 node,
it allows users to run their own node and use the Waku v2 protocols they need.
The nim-waku project doubles as a library, that can be used to add Waku v2 support to native applications.
## Waku v2 in the browser
We believe that dApps and wallets can benefit from the Waku network in several ways.
For some dApps, it makes sense to enable peer-to-peer communications.
For others, machine-to-machine communications would be a great asset.
For example, in the case of a DAO,
Waku could be used for gas-less voting.
Enabling the DAO to notify their users of a new vote,
and users to vote without interacting with the blockchain and spending gas.
[Murmur](https://github.com/status-im/murmur) was the first attempt to bring Whisper to the browser,
acting as a bridge between devp2p and libp2p.
Once Waku v2 was started and there was a native implementation on top of libp2p,
a [chat POC](https://github.com/vacp2p/waku-web-chat) was created to demonstrate the potential of Waku v2
in web environment.
It showed how using js-libp2p with few modifications enabled access to the Waku v2 network.
There was still some unresolved challenges.
For example, nim-waku only support TCP connections which are not supported by browser applications.
Hence, to connect to other node, the POC was connecting to a NodeJS proxy application using websockets,
which in turn could connect to wakunode2 via TCP.
However, to enable dApp and Wallet developers to easily integrate Waku in their product,
we need to give them a library that is easy to use and works out of the box:
introducing [JS-Waku](https://github.com/status-im/js-waku).
JS-Waku is a JavaScript library that allows your dApp, wallet or other web app to interact with the Waku v2 network.
It is available right now on [npm](https://www.npmjs.com/package/js-waku):
`npm install js-waku`.
As it is written in TypeScript, types are included in the npm package to allow easy integration with TypeScript, ClojureScript and other typed languages that compile to JavaScript.
Key Waku v2 protocols are already available:
[message](https://rfc.vac.dev/spec/14/), [store](https://rfc.vac.dev/spec/13/), [relay](https://rfc.vac.dev/spec/11/) and [light push](https://rfc.vac.dev/spec/19/),
enabling your dApp to:
- Send and receive near-instant messages on the Waku network (relay),
- Query nodes for messages that may have been missed, e.g. due to poor cellular network (store),
- Send messages with confirmations (light push).
JS-Waku needs to operate in the same context from which Waku v2 was born:
a restricted environment were connectivity or uptime are not guaranteed;
JS-Waku brings Waku v2 to the browser.
## Achievements so far
We focused the past month on developing a [ReactJS Chat App](https://status-im.github.io/js-waku/).
The aim was to create enough building blocks in JS-Waku to enable this showcase web app that
we now [use for dogfooding](https://github.com/status-im/nim-waku/issues/399) purposes.
Most of the effort was on getting familiar with the [js-libp2p](https://github.com/libp2p/js-libp2p) library
that we heavily rely on.
JS-Waku is the second implementation of Waku v2 protocol,
so a lot of effort on interoperability was needed.
For example, to ensure compatibility with the nim-waku reference implementation,
we run our [tests against wakunode2](https://github.com/status-im/js-waku/blob/90c90dea11dfd1277f530cf5d683fb92992fe141/src/lib/waku_relay/index.spec.ts#L137) as part of the CI.
This interoperability effort helped solidify the current Waku v2 specifications:
By clarifying the usage of topics
([#327](https://github.com/vacp2p/rfc/issues/327), [#383](https://github.com/vacp2p/rfc/pull/383)),
fix discrepancies between specs and nim-waku
([#418](https://github.com/status-im/nim-waku/issues/418), [#419](https://github.com/status-im/nim-waku/issues/419))
and fix small nim-waku & nim-libp2p bugs
([#411](https://github.com/status-im/nim-waku/issues/411), [#439](https://github.com/status-im/nim-waku/issues/439)).
To fully access the waku network, JS-Waku needs to enable web apps to connect to nim-waku nodes.
A standard way to do so is using secure websockets as it is not possible to connect directly to a TCP port from the browser.
Unfortunately websocket support is not yet available in [nim-libp2p](https://github.com/status-im/nim-libp2p/issues/407) so
we ended up deploying [websockify](https://github.com/novnc/websockify) alongside wakunode2 instances.
As we built the [web chat app](https://github.com/status-im/js-waku/tree/main/examples/web-chat),
we were able to fine tune the API to provide a simple and succinct interface.
You can start a node, connect to other nodes and send a message in less than ten lines of code:
```javascript
import { Waku } from "js-waku";
const waku = await Waku.create({});
const nodes = await getStatusFleetNodes();
await Promise.all(nodes.map((addr) => waku.dial(addr)));
const msg = WakuMessage.fromUtf8String(
"Here is a message!",
"/my-cool-app/1/my-use-case/proto"
);
await waku.relay.send(msg);
```
We have also put a bounty at [0xHack](https://0xhack.dev/) for using JS-Waku
and running a [workshop](https://vimeo.com/551509621).
We were thrilled to have a couple of hackers create new software using our libraries.
One of the projects aimed to create a decentralised, end-to-end encrypted messenger app,
similar to what the [ETH-DM](https://rfc.vac.dev/spec/20/) protocol aims to achieve.
Another project was a decentralised Twitter platform.
Such projects allow us to prioritize the work on JS-Waku and understand how DevEx can be improved.
As more developers use JS-Waku, we will evolve the API to allow for more custom and fine-tune usage of the network
while preserving this out of the box experience.
## What's next?
Next, we are directing our attention towards [Developer Experience](https://github.com/status-im/js-waku/issues/68).
We already have [documentation](https://www.npmjs.com/package/js-waku) available but we want to provide more:
[Tutorials](https://github.com/status-im/js-waku/issues/56), various examples
and showing how [JS-Waku can be used with Web3](https://github.com/status-im/js-waku/issues/72).
By prioritizing DevEx we aim to enable JS-Waku integration in dApps and wallets.
We think JS-Waku builds a strong case for machine-to-machine (M2M) communications.
The first use cases we are looking into are dApp notifications:
Enabling dApp to notify their user directly in their wallets!
Leveraging Waku as a decentralised infrastructure and standard so that users do not have to open their dApp to be notified
of events such as DAO voting.
We already have some POC in the pipeline to enable voting and polling on the Waku network,
allowing users to save gas by **not** broadcasting each individual vote on the blockchain.
To facilitate said applications, we are looking at improving integration with Web3 providers by providing examples
of signing, validating, encrypting and decrypting messages using Web3.
Waku is privacy conscious, so we will also provide signature and encryption examples decoupled from users' Ethereum identity.
As you can read, we have grand plans for JS-Waku and Waku v2.
There is a lot to do, and we would love some help so feel free to
check out the new role in our team:
[js-waku: Wallet & Dapp Integration Developer](https://status.im/our_team/jobs.html?gh_jid=3157894).
We also have a number of [positions](https://status.im/our_team/jobs.html) open to work on Waku protocol and nim-waku.
If you are as excited as us by JS-Waku, why not build a dApp with it?
You can find documentation on the [npmjs page](https://www.npmjs.com/package/js-waku).
Whether you are a developer, you can come chat with us using [WakuJS Web Chat](https://status-im.github.io/js-waku/)
or [chat2](https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md).
You can get support on Discord [#waku-support (dev support)](https://discord.gg/VChNsDdj).
If you have any ideas on how Waku could enable a specific dapp or use case, do share, we are always keen to hear it.

58
_posts/2020-11-10-waku-v2-ethereum-messaging.md
View File

@ -1,18 +1,18 @@
---
layout: post
name: "[Talk] Vac, Waku v2 and Ethereum Messaging"
title: "[Talk] Vac, Waku v2 and Ethereum Messaging"
date: 2020-11-10 12:00:00 +0800
name: "[Talk] Vac, Waku v2 and Ethereum Messaging"
title: "[Talk] Vac, Waku v2 and Ethereum Messaging"
date: 2020-11-10 12:00:00 +0800
author: oskarth
published: true
permalink: /waku-v2-ethereum-messaging
categories: research
category: research
summary: Talk from Taipei Ethereum Meetup. Read on to find out about our journey from Whisper to Waku v2, as well as how Waku v2 can be useful for Etherum Messaging.
image: /assets/img/taipei_ethereum_meetup_slide.png
discuss: https://forum.vac.dev/t/discussion-talk-vac-waku-v2-and-ethereum-messaging/60
---
*The following post is a transcript of the talk given at the [Taipei Ethereum meetup, November 5](https://www.meetup.com/Taipei-Ethereum-Meetup/events/274033344/). There is also a [video recording]( https://www.youtube.com/watch?v=lUDy1MoeYnI).*
_The following post is a transcript of the talk given at the [Taipei Ethereum meetup, November 5](https://www.meetup.com/Taipei-Ethereum-Meetup/events/274033344/). There is also a [video recording](https://www.youtube.com/watch?v=lUDy1MoeYnI)._
---
@ -68,9 +68,9 @@ Taking a step back, what problem does Waku v2 attempt to solve compared to all t
3. **Resource restricted**. These applications often run in constrained environments, where resources or the environment is restricted in some fashion. E.g.:
- limited bandwidth, CPU, memory, disk, battery, etc
- not being publicly connectable
- only being intermittently connected; mostly-offline
- limited bandwidth, CPU, memory, disk, battery, etc
- not being publicly connectable
- only being intermittently connected; mostly-offline
4. **Privacy**. These applications have a desire for some privacy guarantees, such as pseudonymity, metadata protection in transit, etc.
@ -86,12 +86,12 @@ It is running over libp2p and we had our second internal testnet last week or so
The main implementation is written in Nim using nim-libp2p, which is also powering Nimbus, an Ethereum 2 client. There is also a PoC for running Waku v2 in the browser. On a spec level, we have the following specifications that corresponds to the components that make up Waku v2:
- Waku v2 - this is the main spec that explains the goals of providing generalized messaging, in a p2p context, with a focus on privacy and running on resources restricted devices.
- Relay - this is the main PubSub spec that provides better routing. It builds on top of GossipSub, which is what Eth2 heavily relies on as well.
- Store - this is a 1-1 protocol for light nodes to get historical messages, if they are mostly-offline.
- Filter - this is a 1-1 protocol for light nodes that are bandwidth restricted to only (or mostly) get messages they care about.
- Message - this explains the payload, to get some basic encryption and content topics. It corresponds roughly to envelopes in Whisper/Waku v1.
- Bridge - this explains how to do bridging between Waku v1 and Waku v2 for compatibility.
- Waku v2 - this is the main spec that explains the goals of providing generalized messaging, in a p2p context, with a focus on privacy and running on resources restricted devices.
- Relay - this is the main PubSub spec that provides better routing. It builds on top of GossipSub, which is what Eth2 heavily relies on as well.
- Store - this is a 1-1 protocol for light nodes to get historical messages, if they are mostly-offline.
- Filter - this is a 1-1 protocol for light nodes that are bandwidth restricted to only (or mostly) get messages they care about.
- Message - this explains the payload, to get some basic encryption and content topics. It corresponds roughly to envelopes in Whisper/Waku v1.
- Bridge - this explains how to do bridging between Waku v1 and Waku v2 for compatibility.
Right now, all protocols, with the exception of bridge, are in draft mode, meaning they have been implemented but are not yet being relied upon in production.
@ -109,10 +109,10 @@ There are also some changes we made to at what level content encryption happens,
More generally, beyond getting to production-ready use, there are a few bigger pieces that we are working on or will work on soon. These are things like:
- Better scaling, by using topic sharding.
- Accounting and user-run nodes, to account for and incentives full nodes.
- Stronger and more rigorous privacy guarantees, e.g. through study of GossipSub, unlinkable packet formats, etc.
- Rate Limit Nullifier for privacy preserving spam protection, a la what Barry Whitehat has presented before.
- Better scaling, by using topic sharding.
- Accounting and user-run nodes, to account for and incentives full nodes.
- Stronger and more rigorous privacy guarantees, e.g. through study of GossipSub, unlinkable packet formats, etc.
- Rate Limit Nullifier for privacy preserving spam protection, a la what Barry Whitehat has presented before.
As well as better support for Ethereum M2M Messaging. Which is what I'll talk about next.
@ -126,12 +126,12 @@ It is easy to think that Waku v2 is only for human to human messaging, since tha
What is Ethereum M2M messaging? Going back to the Holy Trinity of Ethereum/Whisper/Swarm, the messaging component was seen as something that could facilitate messages between dapps and acts as a building block. This can help with things such as:
- Reducing on-chain transactions
- Reduce latency for operations
- Decentralize centrally coordinated services (like WalletConnect)
- Improve UX of dapps
- Broadcast live information
- A message transport layer for state channels
- Reducing on-chain transactions
- Reduce latency for operations
- Decentralize centrally coordinated services (like WalletConnect)
- Improve UX of dapps
- Broadcast live information
- A message transport layer for state channels
And so on.
@ -139,11 +139,11 @@ And so on.
What are some examples of practical things Waku as used for Ethereum Messaging could solve?
- Multisig transfers only needing one on chain transaction
- DAO votes only needing one one chain transaction
- Giving dapps ability to direct push notifications to users
- Giving users ability to directly respond to requests from daps
- Decentralized Wallet Connect
- Multisig transfers only needing one on chain transaction
- DAO votes only needing one one chain transaction
- Giving dapps ability to direct push notifications to users
- Giving users ability to directly respond to requests from daps
- Decentralized Wallet Connect
Etc.

148
_posts/2021-03-03-rln-relay.md
View File

@ -1,20 +1,23 @@
---
layout: post
name: "Privacy-preserving p2p economic spam protection in Waku v2"
title: "Privacy-preserving p2p economic spam protection in Waku v2"
date: 2021-03-05 12:00:00 +0800
name: "Privacy-preserving p2p economic spam protection in Waku v2"
title: "Privacy-preserving p2p economic spam protection in Waku v2"
date: 2021-03-05 12:00:00 +0800
author: sanaztaheri
published: true
permalink: /rln-relay
categories: research
category: research
summary: This post is going to give you an overview of how spam protection can be achieved in Waku Relay through rate-limiting nullifiers. We will cover a summary of spam-protection methods in centralized and p2p systems, and the solution overview and details of the economic spam-protection method. The open issues and future steps are discussed in the end.
image: /assets/img/rain.png
discuss: https://forum.vac.dev/t/privacy-preserving-p2p-economic-spam-protection-in-waku-v2-with-rate-limiting-nullfiers/66
---
# Introduction
This post is going to give you an overview of how spam protection can be achieved in Waku Relay protocol[^2] through Rate-Limiting Nullifiers[^3] [^4] or RLN for short.
Let me give a little background about Waku(v2)[^1]. Waku is a privacy-preserving peer-to-peer (p2p) messaging protocol for resource-restricted devices. Being p2p means that Waku relies on **No** central server. Instead, peers collaboratively deliver messages in the network. Waku uses GossipSub[^16] as the underlying routing protocol (as of the writeup of this post). At a high level, GossipSub is based on publisher-subscriber architecture. That is, *peers, congregate around topics they are interested in and can send messages to topics. Each message gets delivered to all peers subscribed to the topic*. In GossipSub, a peer has a constant number of direct connections/neighbors. In order to publish a message, the author forwards its message to a subset of neighbors. The neighbors proceed similarly till the message gets propagated in the network of the subscribed peers. The message publishing and routing procedures are part of the Waku Relay[^17] protocol.
Let me give a little background about Waku(v2)[^1]. Waku is a privacy-preserving peer-to-peer (p2p) messaging protocol for resource-restricted devices. Being p2p means that Waku relies on **No** central server. Instead, peers collaboratively deliver messages in the network. Waku uses GossipSub[^16] as the underlying routing protocol (as of the writeup of this post). At a high level, GossipSub is based on publisher-subscriber architecture. That is, _peers, congregate around topics they are interested in and can send messages to topics. Each message gets delivered to all peers subscribed to the topic_. In GossipSub, a peer has a constant number of direct connections/neighbors. In order to publish a message, the author forwards its message to a subset of neighbors. The neighbors proceed similarly till the message gets propagated in the network of the subscribed peers. The message publishing and routing procedures are part of the Waku Relay[^17] protocol.
<p align="center">
<img src="../assets/img/rln-relay/rln-relay-overview.png" width="200%" />
<br />
@ -22,54 +25,70 @@ Let me give a little background about Waku(v2)[^1]. Waku is a privacy-preserving
</p>
## What do we mean by spamming?
In centralized messaging systems, a spammer usually indicates an entity that uses the messaging system to send an unsolicited message (spam) to large numbers of recipients. However, in Waku with a p2p architecture, spam messages not only affect the recipients but also all the other peers involved in the routing process as they have to spend their computational power/bandwidth/storage capacity on processing spam messages. As such, we define a spammer as an entity that uses the messaging system to publish a large number of messages in a short amount of time. The messages issued in this way are called spam. In this definition, we disregard the intention of the spammer as well as the content of the message and the number of recipients.
## Possible Solutions
Has the spamming issue been addressed before? Of course yes! Here is an overview of the spam protection techniques with their trade-offs and use-cases. In this overview, we distinguish between protection techniques that are targeted for centralized messaging systems and those for p2p architectures.
### Centralized Messaging Systems
In traditional centralized messaging systems, spam usually signifies unsolicited messages sent in bulk or messages with malicious content like malware. Protection mechanisms include
In traditional centralized messaging systems, spam usually signifies unsolicited messages sent in bulk or messages with malicious content like malware. Protection mechanisms include
- authentication through some piece of personally identifiable information e.g., phone number
- checksum-based filtering to protect against messages sent in bulk
- challenge-response systems
- content filtering on the server or via a proxy application
These methods exploit the fact that the messaging system is centralized and a global view of the users' activities is available based on which spamming patterns can be extracted and defeated accordingly. Moreover, users are associated with an identifier e.g., a username which enables the server to profile each user e.g., to detect suspicious behavior like spamming. Such profiling possibility is against the user's anonymity and privacy.
These methods exploit the fact that the messaging system is centralized and a global view of the users' activities is available based on which spamming patterns can be extracted and defeated accordingly. Moreover, users are associated with an identifier e.g., a username which enables the server to profile each user e.g., to detect suspicious behavior like spamming. Such profiling possibility is against the user's anonymity and privacy.
Among the techniques enumerated above, authentication through phone numbers is a some-what economic-incentive measure as providing multiple valid phone numbers will be expensive for the attacker. Notice that while using an expensive authentication method can reduce the number of accounts owned by a single spammer, cannot address the spam issue entirely. This is because the spammer can still send bulk messages through one single account. For this approach to be effective, a centralized mediator is essential. That is why such a solution would not fit the p2p environments where no centralized control exists.
### P2P Systems
What about spam prevention in p2p messaging platforms? There are two techniques, namely *Proof of Work*[^8] deployed by Whisper[^9] and *Peer scoring*[^6] method (namely reputation-based approach) adopted by LibP2P. However, each of these solutions has its own shortcomings for real-life use-cases as explained below.
What about spam prevention in p2p messaging platforms? There are two techniques, namely _Proof of Work_[^8] deployed by Whisper[^9] and _Peer scoring_[^6] method (namely reputation-based approach) adopted by LibP2P. However, each of these solutions has its own shortcomings for real-life use-cases as explained below.
#### Proof of work
The idea behind the Proof Of Work i.e., POW[^8] is to make messaging a computationally costly operation hence lowering the messaging rate of **all** the peers including the spammers. In specific, the message publisher has to solve a puzzle and the puzzle is to find a nonce such that the hash of the message concatenated with the nonce has at least z leading zeros. z is known as the difficulty of the puzzle. Since the hash function is one-way, peers have to brute-force to find a nonce. Hashing is a computationally-heavy operation so is the brute-force. While solving the puzzle is computationally expensive, it is comparatively cheap to verify the solution.
POW is also used as the underlying mining algorithm in Ethereum and Bitcoin blockchain. There, the goal is to contain the mining speed and allow the decentralized network to come to a consensus, or agree on things like account balances and the order of transactions.
POW is also used as the underlying mining algorithm in Ethereum and Bitcoin blockchain. There, the goal is to contain the mining speed and allow the decentralized network to come to a consensus, or agree on things like account balances and the order of transactions.
While the use of POW makes perfect sense in Ethereum / Bitcoin blockchain, it shows practical issues in heterogeneous p2p messaging systems with resource-restricted peers. Some peers won't be able to carry the designated computation and will be effectively excluded. Such exclusion showed to be practically an issue in applications like Status, which used to rely on POW for spam-protection, to the extent that the difficulty level had to be set close to zero.
While the use of POW makes perfect sense in Ethereum / Bitcoin blockchain, it shows practical issues in heterogeneous p2p messaging systems with resource-restricted peers. Some peers won't be able to carry the designated computation and will be effectively excluded. Such exclusion showed to be practically an issue in applications like Status, which used to rely on POW for spam-protection, to the extent that the difficulty level had to be set close to zero.
#### Peer Scoring
The peer scoring method[^6] that is utilized by libp2p is to limit the number of messages issued by a peer in connection to another peer. That is each peer monitors all the peers to which it is directly connected and adjusts their messaging quota i.e., to route or not route their messages depending on their past activities. For example, if a peer detects its neighbor is sending more than x messages per month, can drop its quota to z.x where z is less than one. The shortcoming of this solution is that scoring is based on peers' local observations and the concept of the score is defined in relation to one single peer. This leaves room for an attack where a spammer can make connections to k peers in the system and publishes k.(x-1) messages by exploiting all of its k connections. Another attack scenario is through botnets consisting of a large number of e.g., a million bots. The attacker rents a botnet and inserts each of them as a legitimate peer to the network and each can publish x-1 messages per month[^7].
The peer scoring method[^6] that is utilized by libp2p is to limit the number of messages issued by a peer in connection to another peer. That is each peer monitors all the peers to which it is directly connected and adjusts their messaging quota i.e., to route or not route their messages depending on their past activities. For example, if a peer detects its neighbor is sending more than x messages per month, can drop its quota to z.x where z is less than one. The shortcoming of this solution is that scoring is based on peers' local observations and the concept of the score is defined in relation to one single peer. This leaves room for an attack where a spammer can make connections to k peers in the system and publishes k.(x-1) messages by exploiting all of its k connections. Another attack scenario is through botnets consisting of a large number of e.g., a million bots. The attacker rents a botnet and inserts each of them as a legitimate peer to the network and each can publish x-1 messages per month[^7].
#### Economic-Incentive Spam protection
Is this the end of our spam-protection journey? Shall we simply give up and leave spammers be? Certainly not!
Waku RLN-Relay gives us a p2p spam-protection method which:
- suits **p2p** systems and does not rely on any central entity.
- is **efficient** i.e., with no unreasonable computational, storage, memory, and bandwidth requirement! as such, it fits the network of **heterogeneous** peers.
- is **efficient** i.e., with no unreasonable computational, storage, memory, and bandwidth requirement! as such, it fits the network of **heterogeneous** peers.
- respects users **privacy** unlike reputation-based and centralized methods.
- deploys **economic-incentives** to contain spammers' activity. Namely, there is a financial sacrifice for those who want to spam the system. How? follow along ...
We devise a general rule to save everyone's life and that is
We devise a general rule to save everyone's life and that is
**No one can publish more than M messages per epoch without being financially charged!**
**No one can publish more than M messages per epoch without being financially charged!**
We set M to 1 for now, but this can be any arbitrary value. You may be thinking "This is too restrictive! Only one per epoch?". Don't worry, we set the epoch to a reasonable value so that it does not slow down the communication of innocent users but will make the life of spammers harder! Epoch here can be every second, as defined by UTC date-time +-20s.
The remainder of this post is all about the story of how to enforce this limit on each user's messaging rate as well as how to impose the financial cost when the limit gets violated. This brings us to the Rate Limiting Nullifiers and how we integrate this technique into Waku v2 (in specific the Waku Relay protocol) to protect our valuable users against spammers.
The remainder of this post is all about the story of how to enforce this limit on each user's messaging rate as well as how to impose the financial cost when the limit gets violated. This brings us to the Rate Limiting Nullifiers and how we integrate this technique into Waku v2 (in specific the Waku Relay protocol) to protect our valuable users against spammers.
# Technical Terms
**Zero-knowledge proof**: Zero-knowledge proof (ZKP)[^14] allows a *prover* to show a *verifier* that they know something, without revealing what that something is. This means you can do the trust-minimized computation that is also privacy-preserving. As a basic example, instead of showing your ID when going to a bar you simply give them proof that you are over 18, without showing the doorman your id. In this write-up, by ZKP we essentially mean zkSNARK[^15] which is one of the many types of ZKPs.
**Zero-knowledge proof**: Zero-knowledge proof (ZKP)[^14] allows a _prover_ to show a _verifier_ that they know something, without revealing what that something is. This means you can do the trust-minimized computation that is also privacy-preserving. As a basic example, instead of showing your ID when going to a bar you simply give them proof that you are over 18, without showing the doorman your id. In this write-up, by ZKP we essentially mean zkSNARK[^15] which is one of the many types of ZKPs.
**Threshold Secret Sharing Scheme**: (m,n) Threshold secret-sharing is a method by which you can split a secret value s into n pieces in a way that the secret s can be reconstructed by having m pieces (m <= n). The economic-incentive spam protection utilizes a (2,n) secret sharing realized by Shamir Secret Sharing Scheme[^13].
# Overview: Economic-Incentive Spam protection through Rate Limiting Nullifiers
**Context**: We started the idea of economic-incentive spam protection more than a year ago and conducted a feasibility study to identify blockers and unknowns. The results are published in our prior [post](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks). Since then major progress has been made and the prior identified blockers that are listed below are now addressed. Kudos to [Barry WhiteHat](https://github.com/barryWhiteHat), [Onur Kilic](https://github.com/kilic), [Koh Wei Jie](https://github.com/weijiekoh/perpetualpowersoftau) for all of their hard work, research, and development which made this progress possible.
- the proof time[^22] which was initially in the order of minutes ~10 mins and now is almost 0.5 seconds
- the prover key size[^21] which was initially ~110MB and now is ~3.9MB
- the lack of Shamir logic[^19] which is now implemented and part of the RLN repository[^4]
@ -79,20 +98,24 @@ The remainder of this post is all about the story of how to enforce this limit o
Now that you have more context, let's see how the final solution works. The fundamental point is to make it economically costly to send more than your share of messages and to do so in a privacy-preserving and e2e fashion. To do that we have the following components:
- 1- **Group**: We manage all the peers inside a large group (later we can split peers into smaller groups, but for now consider only one). The group management is done via a smart contract which is devised for this purpose and is deployed on the Ethereum blockchain.
- 2- **Membership**: To be able to send messages and in specific for the published messages to get routed by all the peers, publishing peers have to register to the group. Membership involves setting up public and private key pairs (think of it as the username and password). The private key remains at the user side but the public key becomes a part of the group information on the contract (publicly available) and everyone has access to it. Public keys are not human-generated (like usernames) and instead they are random numbers, as such, they do not reveal any information about the owner (think of public keys as pseudonyms). Registration is mandatory for the users who want to publish a message, however, users who only want to listen to the messages are more than welcome and do not have to register in the group.
- **Membership fee**: Membership is not for free! each peer has to lock a certain amount of funds during the registration (this means peers have to have an Ethereum account with sufficient balance for this sake). This fund is safely stored on the contract and remains intact unless the peer attempts to break the rules and publish more than one message per epoch.
- 2- **Membership**: To be able to send messages and in specific for the published messages to get routed by all the peers, publishing peers have to register to the group. Membership involves setting up public and private key pairs (think of it as the username and password). The private key remains at the user side but the public key becomes a part of the group information on the contract (publicly available) and everyone has access to it. Public keys are not human-generated (like usernames) and instead they are random numbers, as such, they do not reveal any information about the owner (think of public keys as pseudonyms). Registration is mandatory for the users who want to publish a message, however, users who only want to listen to the messages are more than welcome and do not have to register in the group.
- **Membership fee**: Membership is not for free! each peer has to lock a certain amount of funds during the registration (this means peers have to have an Ethereum account with sufficient balance for this sake). This fund is safely stored on the contract and remains intact unless the peer attempts to break the rules and publish more than one message per epoch.
- **Zero-knowledge Proof of membership**: Do you want your message to get routed to its destination, fine, but you have to prove that you are a member of the group (sorry, no one can escape the registration phase!). Now, you may be thinking that should I attach my public key to my message to prove my membership? Absolutely Not! we said that our solution respects privacy! membership proofs are done in a zero-knowledge manner that is each message will carry cryptographic proof asserting that "the message is generated by one of the current members of the group", so your identity remains private and your anonymity is preserved!
- **Slashing through secret sharing**: Till now it does not seem like we can catch spammers, right? yes, you are right! now comes the exciting part, detecting spammers and slashing them. The core idea behind the slashing is that each publishing peer (not routing peers!) has to integrate a secret share of its private key inside the message. The secret share is deterministically computed over the private key and the current epoch. The content of this share is harmless for the peer's privacy (it looks random) unless the peer attempts to publish more than one message in the same epoch hence disclosing more than one secret share of its private key. Indeed two distinct shares of the private key under the same epoch are enough to reconstruct the entire private key. Then what should you do with the recovered private key? hurry up! go to the contract and withdraw the private key and claim its fund and get rich!! Are you thinking what if spammers attach junk values instead of valid secret shares? Of course, that wouldn't be cool! so, there is a zero-knowledge proof for this sake as well where the publishing peer has to prove that the secret shares are generated correctly.
A high-level overview of the economic spam protection is shown in Figure 1.
# Flow
# Flow
In this section, we describe the flow of the economic-incentive spam detection mechanism from the viewpoint of a single peer. An overview of this flow is provided in Figure 3.
## Setup and Registration
A peer willing to publish a message is required to register. Registration is moderated through a smart contract deployed on the Ethereum blockchain. The state of the contract contains the list of registered members' public keys. An overview of registration is illustrated in Figure 2.
For the registration, a peer creates a transaction that sends x amount of Ether to the contract. The peer who has the "private key" `sk` associated with that deposit would be able to withdraw x Ether by providing valid proof. Note that `sk` is initially only known by the owning peer however it may get exposed to other peers in case the owner attempts spamming the system i.e., sending more than one message per epoch.
For the registration, a peer creates a transaction that sends x amount of Ether to the contract. The peer who has the "private key" `sk` associated with that deposit would be able to withdraw x Ether by providing valid proof. Note that `sk` is initially only known by the owning peer however it may get exposed to other peers in case the owner attempts spamming the system i.e., sending more than one message per epoch.
The following relation holds between the `sk` and `pk` i.e., `pk = H(sk)` where `H` denotes a hash function.
<p align="center">
<img src="../assets/img/rln-relay/rln-relay.png" />
<br />
@ -100,38 +123,46 @@ The following relation holds between the `sk` and `pk` i.e., `pk = H(sk)` where
</p>
## Maintaining the membership Merkle Tree
The ZKP of membership that we mentioned before relies on the representation of the entire group as a [Merkle Tree](). The tree construction and maintenance is delegated to the peers (the initial idea was to keep the tree on the chain as part of the contract, however, the cost associated with member deletion and insertion was high and unreasonable, please see [Feasibility and Open Issues](#Feasibility-and-Open-Issues) for more details). As such, each peer needs to build the tree locally and sync itself with the contract updates (peer insertion and deletion) to mirror them on its tree.
Two pieces of information of the tree are important as they enable peers to generate zero-knowledge proofs. One is the root of the tree and the other is the membership proof (or the authentication path). The tree root is public information whereas the membership proof is private data (or more precisely the index of the peer in the tree).
Two pieces of information of the tree are important as they enable peers to generate zero-knowledge proofs. One is the root of the tree and the other is the membership proof (or the authentication path). The tree root is public information whereas the membership proof is private data (or more precisely the index of the peer in the tree).
## Publishing
In order to publish at a given epoch, each message must carry a proof i.e., a zero-knowledge proof signifying that the publishing peer is a registered member, and has not exceeded the messaging rate at the given epoch.
In order to publish at a given epoch, each message must carry a proof i.e., a zero-knowledge proof signifying that the publishing peer is a registered member, and has not exceeded the messaging rate at the given epoch.
Recall that the enforcement of the messaging rate was through associating a secret shared version of the peer's `sk` into the message together with a ZKP that the secret shares are constructed correctly. As for the secret sharing part, the peer generates the following data:
1. `shareX`
2. `shareY`
3. `nullifier`
The pair (`shareX`, `shareY`) is the secret shared version of `sk` that are generated using Shamir secret sharing scheme. Having two such pairs for an identical `nullifier` results in full disclosure of peer's `sk` and hence burning the associated deposit. Note that the `nullifier` is a deterministic value derived from `sk` and `epoch` therefore any two messages issued by the same peer (i.e., using the same `sk`) for the same `epoch` are guaranteed to have identical `nullifier`s.
The pair (`shareX`, `shareY`) is the secret shared version of `sk` that are generated using Shamir secret sharing scheme. Having two such pairs for an identical `nullifier` results in full disclosure of peer's `sk` and hence burning the associated deposit. Note that the `nullifier` is a deterministic value derived from `sk` and `epoch` therefore any two messages issued by the same peer (i.e., using the same `sk`) for the same `epoch` are guaranteed to have identical `nullifier`s.
Finally, the peer generates a zero-knowledge proof `zkProof` asserting the membership of the peer in the group and the correctness of the attached secret share (`shareX`, `shareY`) and the `nullifier`. In order to generate a valid proof, the peer needs to have two private inputs i.e., its `sk` and its authentication path. Other inputs are the tree root, epoch, and the content of the message.
Finally, the peer generates a zero-knowledge proof `zkProof` asserting the membership of the peer in the group and the correctness of the attached secret share (`shareX`, `shareY`) and the `nullifier`. In order to generate a valid proof, the peer needs to have two private inputs i.e., its `sk` and its authentication path. Other inputs are the tree root, epoch, and the content of the message.
**Privacy Hint:** Note that the authentication path of each peer depends on the recent list of members (hence changes when new peers register or leave). As such, it is recommended (and necessary for privacy/anonymity) that the publisher updates her authentication path based on the latest status of the group and attempts the proof using the updated version.
An overview of the publishing procedure is provided in Figure 3.
## Routing
Upon the receipt of a message, the routing peer needs to decide whether to route it or not. This decision relies on the following factors:
1) If the epoch value attached to the message has a non-reasonable gap with the routing peer's current epoch then the message must be dropped (this is to prevent a newly registered peer spamming the system by messaging for all the past epochs).
2) The message MUST contain valid proof that gets verified by the routing peer.
If the preceding checks are passed successfully, then the message is relayed. In case of an invalid proof, the message is dropped. If spamming is detected, the publishing peer gets slashed (see [Spam Detection and Slashing](#Spam-Detection-and-Slashing)).
1. If the epoch value attached to the message has a non-reasonable gap with the routing peer's current epoch then the message must be dropped (this is to prevent a newly registered peer spamming the system by messaging for all the past epochs).
2. The message MUST contain valid proof that gets verified by the routing peer.
If the preceding checks are passed successfully, then the message is relayed. In case of an invalid proof, the message is dropped. If spamming is detected, the publishing peer gets slashed (see [Spam Detection and Slashing](#Spam-Detection-and-Slashing)).
An overview of the routing procedure is provided in Figure 3.
### Spam Detection and Slashing
In order to enable local spam detection and slashing, routing peers MUST record the `nullifier`, `shareX`, and `shareY` of any incoming message conditioned that it is not spam and has valid proof. To do so, the peer should follow the following steps.
1. The routing peer first verifies the `zkProof` and drops the message if not verified.
2. Otherwise, it checks whether a message with an identical `nullifier` has already been relayed.
+ a) If such message exists and its `shareX` and `shareY` components are different from the incoming message, then slashing takes place (if the `shareX` and `shareY` fields of the previously relayed message is identical to the incoming message, then the message is a duplicate and shall be dropped).
+ b) If none found, then the message gets relayed.
In order to enable local spam detection and slashing, routing peers MUST record the `nullifier`, `shareX`, and `shareY` of any incoming message conditioned that it is not spam and has valid proof. To do so, the peer should follow the following steps.
1. The routing peer first verifies the `zkProof` and drops the message if not verified.
2. Otherwise, it checks whether a message with an identical `nullifier` has already been relayed.
- a) If such message exists and its `shareX` and `shareY` components are different from the incoming message, then slashing takes place (if the `shareX` and `shareY` fields of the previously relayed message is identical to the incoming message, then the message is a duplicate and shall be dropped).
- b) If none found, then the message gets relayed.
An overview of the slashing procedure is provided in Figure 3.
@ -141,28 +172,41 @@ An overview of the slashing procedure is provided in Figure 3.
Figure 3: Publishing, Routing and Slashing workflow.
</p>
# Feasibility and Open Issues
We've come a long way since a year ago, blockers resolved, now we have implemented it end-to-end. We learned lot and could identify further issues and unknowns some of which are blocking getting to production. The summary of the identified issues are presented below.
## Storage overhead per peer
Currently, peers are supposed to maintain the entire tree locally and it imposes storage overhead which is linear in the size of the group (see this [issue](https://github.com/vacp2p/research/issues/57)[^11] for more details). One way to cope with this is to use the light-node and full-node paradigm in which only a subset of peers who are more resourceful retain the tree whereas the light nodes obtain the necessary information by interacting with the full nodes. Another way to approach this problem is through a more storage efficient method (as described in this research issue[^12]) where peers store a partial view of the tree instead of the entire tree. Keeping the partial view lowers the storage complexity to O(log(N)) where N is the size of the group. There are still unknown unknowns to this solution, as such, it must be studied further to become fully functional.
## Cost-effective way of member insertion and deletion
Currently, the cost associated with RLN-Relay membership is around 30 USD[^10]. We aim at finding a more cost-effective approach. Please feel free to share with us your solution ideas in this regard in this [issue](https://github.com/vacp2p/research/issues/56).
## Exceeding the messaging rate via multiple registrations
While the economic-incentive solution has an economic incentive to discourage spamming, we should note that there is still **expensive attack(s)**[^23] that a spammer can launch to break the messaging rate limit. That is, the attacker can pay for multiple legit registrations e.g., k, hence being able to publish k messages per epoch. We believe that the higher the membership fee is, the less probable would be such an attack, hence a stronger level of spam-protection can be achieved. Following this argument, the high fee associated with the membership (which we listed above as an open problem) can indeed be contributing to a better protection level.
# Conclusion and Future Steps
As discussed in this post, Waku RLN Relay can achieve a privacy-preserving economic spam protection through rate-limiting nullifiers. The idea is to financially discourage peers from publishing more than one message per epoch. In specific, exceeding the messaging rate results in a financial charge. Those who violate this rule are called spammers and their messages are spam. The identification of spammers does not rely on any central entity. Also, the financial punishment of spammers is cryptographically guaranteed.
In this solution, privacy is guaranteed since: 1) Peers do not have to disclose any piece of personally identifiable information in any phase i.e., neither in the registration nor in the messaging phase 2) Peers can prove that they have not exceeded the messaging rate in a zero-knowledge manner and without leaving any trace to their membership accounts.
Furthermore, all the computations are light hence this solution fits the heterogenous p2p messaging system. Note that the zero-knowledge proof parts are handled through zkSNARKs and the benchmarking result can be found in the RLN benchmark report[^5].
**Future steps**: We are still at the PoC level, and the development is in progress. As our future steps,
- we would like to evaluate the running time associated with the Merkle tree operations. Indeed, the need to locally store Merkle tree on each peer was one of the unknowns discovered during this PoC and yet the concrete benchmarking result in this regard is not available.
- We would also like to pursue our storage-efficient Merkle Tree maintenance solution in order to lower the storage overhead of peers.
- In line with the storage optimization, the full-node light-node structure is another path to follow.
- Another possible improvement is to replace the membership contract with a distributed group management scheme e.g., through distributed hash tables. This is to address possible performance issues that the interaction with the Ethereum blockchain may cause. For example, the registration transactions are subject to delay as they have to be mined before being visible in the state of the membership contract. This means peers have to wait for some time before being able to publish any message.
We've come a long way since a year ago, blockers resolved, now we have implemented it end-to-end. We learned lot and could identify further issues and unknowns some of which are blocking getting to production. The summary of the identified issues are presented below.
## Storage overhead per peer
Currently, peers are supposed to maintain the entire tree locally and it imposes storage overhead which is linear in the size of the group (see this [issue](https://github.com/vacp2p/research/issues/57)[^11] for more details). One way to cope with this is to use the light-node and full-node paradigm in which only a subset of peers who are more resourceful retain the tree whereas the light nodes obtain the necessary information by interacting with the full nodes. Another way to approach this problem is through a more storage efficient method (as described in this research issue[^12]) where peers store a partial view of the tree instead of the entire tree. Keeping the partial view lowers the storage complexity to O(log(N)) where N is the size of the group. There are still unknown unknowns to this solution, as such, it must be studied further to become fully functional.
## Cost-effective way of member insertion and deletion
Currently, the cost associated with RLN-Relay membership is around 30 USD[^10]. We aim at finding a more cost-effective approach. Please feel free to share with us your solution ideas in this regard in this [issue](https://github.com/vacp2p/research/issues/56).
## Exceeding the messaging rate via multiple registrations
While the economic-incentive solution has an economic incentive to discourage spamming, we should note that there is still **expensive attack(s)**[^23] that a spammer can launch to break the messaging rate limit. That is, the attacker can pay for multiple legit registrations e.g., k, hence being able to publish k messages per epoch. We believe that the higher the membership fee is, the less probable would be such an attack, hence a stronger level of spam-protection can be achieved. Following this argument, the high fee associated with the membership (which we listed above as an open problem) can indeed be contributing to a better protection level.
# Conclusion and Future Steps
As discussed in this post, Waku RLN Relay can achieve a privacy-preserving economic spam protection through rate-limiting nullifiers. The idea is to financially discourage peers from publishing more than one message per epoch. In specific, exceeding the messaging rate results in a financial charge. Those who violate this rule are called spammers and their messages are spam. The identification of spammers does not rely on any central entity. Also, the financial punishment of spammers is cryptographically guaranteed.
In this solution, privacy is guaranteed since: 1) Peers do not have to disclose any piece of personally identifiable information in any phase i.e., neither in the registration nor in the messaging phase 2) Peers can prove that they have not exceeded the messaging rate in a zero-knowledge manner and without leaving any trace to their membership accounts.
Furthermore, all the computations are light hence this solution fits the heterogenous p2p messaging system. Note that the zero-knowledge proof parts are handled through zkSNARKs and the benchmarking result can be found in the RLN benchmark report[^5].
**Future steps**: We are still at the PoC level, and the development is in progress. As our future steps,
- we would like to evaluate the running time associated with the Merkle tree operations. Indeed, the need to locally store Merkle tree on each peer was one of the unknowns discovered during this PoC and yet the concrete benchmarking result in this regard is not available.
- We would also like to pursue our storage-efficient Merkle Tree maintenance solution in order to lower the storage overhead of peers.
- In line with the storage optimization, the full-node light-node structure is another path to follow.
- Another possible improvement is to replace the membership contract with a distributed group management scheme e.g., through distributed hash tables. This is to address possible performance issues that the interaction with the Ethereum blockchain may cause. For example, the registration transactions are subject to delay as they have to be mined before being visible in the state of the membership contract. This means peers have to wait for some time before being able to publish any message.
# Acknowledgement
Thanks to Onur Kılıç for his explanation and pointers and for assisting with development and runtime issues. Also thanks to Barry Whitehat for his time and insightful comments. Special thanks to Oskar Thoren for his constructive comments and his guides during the development of this PoC and the writeup of this post.
# References
[^1]: Waku v2: [https://github.com/vacp2p/specs/blob/master/specs/waku/v2/waku-v2.md](https://github.com/vacp2p/specs/blob/master/specs/waku/v2/waku-v2.md)
[^2]: RLN-Relay specifications: [https://github.com/vacp2p/specs/blob/master/specs/waku/v2/waku-rln-relay.md](https://github.com/vacp2p/specs/blob/master/specs/waku/v2/waku-rln-relay.md)
[^3]: RLN documentation: [https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?both](https://hackmd.io/tMTLMYmTR5eynw2lwK9n1w?both)
@ -178,11 +222,11 @@ Thanks to Onur Kılıç for his explanation and pointers and for assisting with
[^13]: Shamir Secret Sharing Scheme: [https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing](https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing)
[^14]: Zero Knowledge Proof: [https://dl.acm.org/doi/abs/10.1145/3335741.3335750](https://dl.acm.org/doi/abs/10.1145/3335741.3335750) and [https://en.wikipedia.org/wiki/Zero-knowledge_proof](https://en.wikipedia.org/wiki/Zero-knowledge_proof)
[^15]: zkSNARKs: [https://link.springer.com/chapter/10.1007/978-3-662-49896-5_11](https://link.springer.com/chapter/10.1007/978-3-662-49896-5_11) and [https://coinpare.io/whitepaper/zcash.pdf](https://coinpare.io/whitepaper/zcash.pdf)
[^16]: GossipSub: [https://docs.libp2p.io/concepts/publish-subscribe/](https://docs.libp2p.io/concepts/publish-subscribe/)
[^16]: GossipSub: [https://docs.libp2p.io/concepts/publish-subscribe/](https://docs.libp2p.io/concepts/publish-subscribe/)
[^17]: Waku Relay: [https://github.com/vacp2p/specs/blob/master/specs/waku/v2/waku-relay.md](https://github.com/vacp2p/specs/blob/master/specs/waku/v2/waku-relay.md)
[^18]: Prior blockers of RLN-Relay: [https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks)
[^18]: Prior blockers of RLN-Relay: [https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks)
[^19]: The lack of Shamir secret sharing in zkSNARKs: [https://github.com/vacp2p/research/issues/10](https://github.com/vacp2p/research/issues/10)
[^20]: The MPC required for zkSNARKs trusted setup: [https://github.com/vacp2p/research/issues/9](https://github.com/vacp2p/research/issues/9)
[^21]: Prover key size: [https://github.com/vacp2p/research/issues/8](https://github.com/vacp2p/research/issues/8)
[^22]: zkSNARKs proof time: [https://github.com/vacp2p/research/issues/7](https://github.com/vacp2p/research/issues/7)
[^23]: Attack on the messaging rate: [https://github.com/vacp2p/specs/issues/251](https://github.com/vacp2p/specs/issues/251)
[^23]: Attack on the messaging rate: [https://github.com/vacp2p/specs/issues/251](https://github.com/vacp2p/specs/issues/251)

178
_posts/2021-03-05-problem-1.md Normal file
View File

@ -0,0 +1,178 @@
---
layout: post
name: "Problem No. 1 - Status ipsum lorem exequoe"
title: "Problem No. 1 - Status ipsum lorem exequoe"
date: 2021-03-05 12:00:00 +0800
author: franck
published: true
permalink: /status-ipsum
category: problem
summary: "Status ipsum lorem exequoe"
image: /assets/img/js-waku-gist.png
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
---
For the past 3 months, we have been working on bringing Waku v2 to the browser.
Our aim is to empower dApps with Waku v2, and it led to the creation of a new library.
We believe now is good time to introduce it!
## Waku v2
First, let's review what Waku v2 is and what problem it is trying to solve.
Waku v2 comes from a need to have a more scalable, better optimised solution for the Status app to achieve decentralised
communications on resource restricted devices (i.e., mobile phones).
The Status chat feature was initially built over Whisper.
However, Whisper has a number of caveats which makes it inefficient for mobile phones.
For example, with Whisper, all devices are receiving all messages which is not ideal for limited data plans.
To remediate this, a Waku mode (then Waku v1), based on devp2p, was introduced.
To further enable web and restricted resource environments, Waku v2 was created based on libp2p.
The migration of the Status chat feature to Waku v2 is currently in progress.
We see the need of such solution in the broader Ethereum ecosystem, beyond Status.
This is why we are building Waku v2 as a decentralised communication platform for all to use and build on.
If you want to read more about Waku v2 and what it aims to achieve,
checkout [What's the Plan for Waku v2?](/waku-v2-plan).
Since last year, we have been busy defining and implementing Waku v2 protocols in [nim-waku](https://github.com/status-im/nim-waku),
from which you can build [wakunode2](https://github.com/status-im/nim-waku#wakunode).
Wakunode2 is an adaptive and modular Waku v2 node,
it allows users to run their own node and use the Waku v2 protocols they need.
The nim-waku project doubles as a library, that can be used to add Waku v2 support to native applications.
## Waku v2 in the browser
We believe that dApps and wallets can benefit from the Waku network in several ways.
For some dApps, it makes sense to enable peer-to-peer communications.
For others, machine-to-machine communications would be a great asset.
For example, in the case of a DAO,
Waku could be used for gas-less voting.
Enabling the DAO to notify their users of a new vote,
and users to vote without interacting with the blockchain and spending gas.
[Murmur](https://github.com/status-im/murmur) was the first attempt to bring Whisper to the browser,
acting as a bridge between devp2p and libp2p.
Once Waku v2 was started and there was a native implementation on top of libp2p,
a [chat POC](https://github.com/vacp2p/waku-web-chat) was created to demonstrate the potential of Waku v2
in web environment.
It showed how using js-libp2p with few modifications enabled access to the Waku v2 network.
There was still some unresolved challenges.
For example, nim-waku only support TCP connections which are not supported by browser applications.
Hence, to connect to other node, the POC was connecting to a NodeJS proxy application using websockets,
which in turn could connect to wakunode2 via TCP.
However, to enable dApp and Wallet developers to easily integrate Waku in their product,
we need to give them a library that is easy to use and works out of the box:
introducing [JS-Waku](https://github.com/status-im/js-waku).
JS-Waku is a JavaScript library that allows your dApp, wallet or other web app to interact with the Waku v2 network.
It is available right now on [npm](https://www.npmjs.com/package/js-waku):
`npm install js-waku`.
As it is written in TypeScript, types are included in the npm package to allow easy integration with TypeScript, ClojureScript and other typed languages that compile to JavaScript.
Key Waku v2 protocols are already available:
[message](https://rfc.vac.dev/spec/14/), [store](https://rfc.vac.dev/spec/13/), [relay](https://rfc.vac.dev/spec/11/) and [light push](https://rfc.vac.dev/spec/19/),
enabling your dApp to:
- Send and receive near-instant messages on the Waku network (relay),
- Query nodes for messages that may have been missed, e.g. due to poor cellular network (store),
- Send messages with confirmations (light push).
JS-Waku needs to operate in the same context from which Waku v2 was born:
a restricted environment were connectivity or uptime are not guaranteed;
JS-Waku brings Waku v2 to the browser.
## Achievements so far
We focused the past month on developing a [ReactJS Chat App](https://status-im.github.io/js-waku/).
The aim was to create enough building blocks in JS-Waku to enable this showcase web app that
we now [use for dogfooding](https://github.com/status-im/nim-waku/issues/399) purposes.
Most of the effort was on getting familiar with the [js-libp2p](https://github.com/libp2p/js-libp2p) library
that we heavily rely on.
JS-Waku is the second implementation of Waku v2 protocol,
so a lot of effort on interoperability was needed.
For example, to ensure compatibility with the nim-waku reference implementation,
we run our [tests against wakunode2](https://github.com/status-im/js-waku/blob/90c90dea11dfd1277f530cf5d683fb92992fe141/src/lib/waku_relay/index.spec.ts#L137) as part of the CI.
This interoperability effort helped solidify the current Waku v2 specifications:
By clarifying the usage of topics
([#327](https://github.com/vacp2p/rfc/issues/327), [#383](https://github.com/vacp2p/rfc/pull/383)),
fix discrepancies between specs and nim-waku
([#418](https://github.com/status-im/nim-waku/issues/418), [#419](https://github.com/status-im/nim-waku/issues/419))
and fix small nim-waku & nim-libp2p bugs
([#411](https://github.com/status-im/nim-waku/issues/411), [#439](https://github.com/status-im/nim-waku/issues/439)).
To fully access the waku network, JS-Waku needs to enable web apps to connect to nim-waku nodes.
A standard way to do so is using secure websockets as it is not possible to connect directly to a TCP port from the browser.
Unfortunately websocket support is not yet available in [nim-libp2p](https://github.com/status-im/nim-libp2p/issues/407) so
we ended up deploying [websockify](https://github.com/novnc/websockify) alongside wakunode2 instances.
As we built the [web chat app](https://github.com/status-im/js-waku/tree/main/examples/web-chat),
we were able to fine tune the API to provide a simple and succinct interface.
You can start a node, connect to other nodes and send a message in less than ten lines of code:
```javascript
import { Waku } from "js-waku";
const waku = await Waku.create({});
const nodes = await getStatusFleetNodes();
await Promise.all(nodes.map((addr) => waku.dial(addr)));
const msg = WakuMessage.fromUtf8String(
"Here is a message!",
"/my-cool-app/1/my-use-case/proto"
);
await waku.relay.send(msg);
```
We have also put a bounty at [0xHack](https://0xhack.dev/) for using JS-Waku
and running a [workshop](https://vimeo.com/551509621).
We were thrilled to have a couple of hackers create new software using our libraries.
One of the projects aimed to create a decentralised, end-to-end encrypted messenger app,
similar to what the [ETH-DM](https://rfc.vac.dev/spec/20/) protocol aims to achieve.
Another project was a decentralised Twitter platform.
Such projects allow us to prioritize the work on JS-Waku and understand how DevEx can be improved.
As more developers use JS-Waku, we will evolve the API to allow for more custom and fine-tune usage of the network
while preserving this out of the box experience.
## What's next?
Next, we are directing our attention towards [Developer Experience](https://github.com/status-im/js-waku/issues/68).
We already have [documentation](https://www.npmjs.com/package/js-waku) available but we want to provide more:
[Tutorials](https://github.com/status-im/js-waku/issues/56), various examples
and showing how [JS-Waku can be used with Web3](https://github.com/status-im/js-waku/issues/72).
By prioritizing DevEx we aim to enable JS-Waku integration in dApps and wallets.
We think JS-Waku builds a strong case for machine-to-machine (M2M) communications.
The first use cases we are looking into are dApp notifications:
Enabling dApp to notify their user directly in their wallets!
Leveraging Waku as a decentralised infrastructure and standard so that users do not have to open their dApp to be notified
of events such as DAO voting.
We already have some POC in the pipeline to enable voting and polling on the Waku network,
allowing users to save gas by **not** broadcasting each individual vote on the blockchain.
To facilitate said applications, we are looking at improving integration with Web3 providers by providing examples
of signing, validating, encrypting and decrypting messages using Web3.
Waku is privacy conscious, so we will also provide signature and encryption examples decoupled from users' Ethereum identity.
As you can read, we have grand plans for JS-Waku and Waku v2.
There is a lot to do, and we would love some help so feel free to
check out the new role in our team:
[js-waku: Wallet & Dapp Integration Developer](https://status.im/our_team/jobs.html?gh_jid=3157894).
We also have a number of [positions](https://status.im/our_team/jobs.html) open to work on Waku protocol and nim-waku.
If you are as excited as us by JS-Waku, why not build a dApp with it?
You can find documentation on the [npmjs page](https://www.npmjs.com/package/js-waku).
Whether you are a developer, you can come chat with us using [WakuJS Web Chat](https://status-im.github.io/js-waku/)
or [chat2](https://github.com/status-im/nim-waku/blob/master/docs/tutorial/chat2.md).
You can get support on Discord [#waku-support (dev support)](https://discord.gg/VChNsDdj).
If you have any ideas on how Waku could enable a specific dapp or use case, do share, we are always keen to hear it.

15
_posts/2021-06-04-presenting-js-waku.md
View File

@ -6,10 +6,10 @@ date: 2021-06-04 12:00:00 +0800
author: franck
published: true
permalink: /presenting-js-waku
categories: platform
category: research
summary: "JS-Waku is bringing Waku v2 to the browser. Learn what we achieved so far and what is next in our pipeline!"
image: /assets/img/js-waku-gist.png
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/81
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/81
---
For the past 3 months, we have been working on bringing Waku v2 to the browser.
@ -61,7 +61,7 @@ It showed how using js-libp2p with few modifications enabled access to the Waku
There was still some unresolved challenges.
For example, nim-waku only support TCP connections which are not supported by browser applications.
Hence, to connect to other node, the POC was connecting to a NodeJS proxy application using websockets,
which in turn could connect to wakunode2 via TCP.
which in turn could connect to wakunode2 via TCP.
However, to enable dApp and Wallet developers to easily integrate Waku in their product,
we need to give them a library that is easy to use and works out of the box:
@ -109,7 +109,7 @@ and fix small nim-waku & nim-libp2p bugs
To fully access the waku network, JS-Waku needs to enable web apps to connect to nim-waku nodes.
A standard way to do so is using secure websockets as it is not possible to connect directly to a TCP port from the browser.
Unfortunately websocket support is not yet available in [nim-libp2p](https://github.com/status-im/nim-libp2p/issues/407) so
Unfortunately websocket support is not yet available in [nim-libp2p](https://github.com/status-im/nim-libp2p/issues/407) so
we ended up deploying [websockify](https://github.com/novnc/websockify) alongside wakunode2 instances.
As we built the [web chat app](https://github.com/status-im/js-waku/tree/main/examples/web-chat),
@ -117,14 +117,17 @@ we were able to fine tune the API to provide a simple and succinct interface.
You can start a node, connect to other nodes and send a message in less than ten lines of code:
```javascript
import { Waku } from 'js-waku';
import { Waku } from "js-waku";
const waku = await Waku.create({});
const nodes = await getStatusFleetNodes();
await Promise.all(nodes.map((addr) => waku.dial(addr)));
const msg = WakuMessage.fromUtf8String("Here is a message!", "/my-cool-app/1/my-use-case/proto")
const msg = WakuMessage.fromUtf8String(
"Here is a message!",
"/my-cool-app/1/my-use-case/proto"
);
await waku.relay.send(msg);
```