Change post category
This commit is contained in:
Maria Rushkova
parent
3b60c4c1c7
commit
967c77e73a
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "P2P Data Sync for Mobile"
|
||||
title: "P2P Data Sync for Mobile"
|
||||
name: 'P2P Data Sync for Mobile'
|
||||
title: 'P2P Data Sync for Mobile'
|
||||
date: 2019-07-19 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /p2p-data-sync-for-mobile
|
||||
categories: research
|
||||
category: research
|
||||
summary: A research log. Reliable and decentralized, pick two.
|
||||
image: /assets/img/mvds_interactive.png
|
||||
---
|
||||
|
|
@ -22,14 +22,17 @@ There are many synchronization protocols out there and I won't go into detail of
|
|||
Why do we want to do p2p sync for mobilephones in the first place? There are three components to that question. One is on the value of decentralization and peer-to-peer, the second is on why we'd want to reliably sync data at all, and finally why mobilephones and other resource restricted devices.
|
||||
|
||||
### Why p2p?
|
||||
|
||||
For decentralization and p2p, there are both technical and social/philosophical reasons. Technically, having a user-run network means it can scale with the number of users. Data locality is also improved if you query data that's close to you, similar to distributed CDNs. The throughput is also improved if there are more places to get data from.
|
||||
|
||||
Socially and philosophically, there are several ways to think about it. Open and decentralized networks also relate to the idea of open standards, i.e. compare the longevity of AOL with IRC or Bittorrent. One is run by a company and is shut down as soon as it stops being profitable, the others live on. Additionally increasingly control of data and infrastructure is becoming a liability. By having a network with no one in control, everyone is. It's ultimately a form of democratization, more similar to organic social structures pre Big Internet companies. This leads to properties such as censorship resistance and coercion resistance, where we limit the impact a 3rd party might have a voluntary interaction between individuals or a group of people. Examples of this are plentiful in the world of Facebook, Youtube, Twitter and WeChat.
|
||||
|
||||
### Why reliably sync data?
|
||||
|
||||
At risk of stating the obvious, reliably syncing data is a requirement for many problem domains. You don't get this by default in a p2p world, as it is unreliable with nodes permissionslessly join and leave the network. In some cases you can get away with only ephemeral data, but usually you want some kind of guarantees. This is a must for reliable group chat experience, for example, where messages are expected to arrive in a timely fashion and in some reasonable order. The same is true for messages there represent financial transactions, and so on.
|
||||
|
||||
### Why mobilephones?
|
||||
|
||||
Most devices people use daily are mobile phones. It's important to provide the same or at least similar guarantees to more traditional p2p nodes that might run on a desktop computer or computer. The alternative is to rely on gateways, which shares many of the drawbacks of centralized control and prone to censorship, control and surveillence.
|
||||
|
||||
More generally, resource restricted devices can differ in their capabilities. One example is smartphones, but others are: desktop, routers, Raspberry PIs, POS systems, and so on. The number and diversity of devices are exploding, and it's useful to be able to leverage this for various types of infrastructure. The alternative is to centralize on big cloud providers, which also lends itself to lack of democratization and censorship, etc.
|
||||
|
|
@ -90,15 +93,16 @@ Each failed attempt implies another retransmission. That means we have `(1/0.1)^
|
|||
|
||||
The problem above hints at the requirements 3 and 4 above. While we did get reliable syncing (requirement 1), it came at a big cost.
|
||||
|
||||
There are a few ways of getting around this issue. One is having a *store and forward* model, where some intermediary node picks up (encrypted) messages and forwards them to the recipient. This is what we have in production right now at Status.
|
||||
There are a few ways of getting around this issue. One is having a _store and forward_ model, where some intermediary node picks up (encrypted) messages and forwards them to the recipient. This is what we have in production right now at Status.
|
||||
|
||||
Another, arguably more pure and robust, way is having a *remote log*, where the actual data is spread over some decentralized storage layer, and you have a mutable reference to find the latest messages, similar to DNS.
|
||||
Another, arguably more pure and robust, way is having a _remote log_, where the actual data is spread over some decentralized storage layer, and you have a mutable reference to find the latest messages, similar to DNS.
|
||||
|
||||
What they both have in common is that they act as a sort of highly-available cache to smooth over the non-overlapping connection windows between two endpoints. Neither of them are *required* to get reliable data transmission.
|
||||
What they both have in common is that they act as a sort of highly-available cache to smooth over the non-overlapping connection windows between two endpoints. Neither of them are _required_ to get reliable data transmission.
|
||||
|
||||
### Basic calculations for bandwidth multiplier
|
||||
|
||||
While we do want better simulations, and this is a work in progress, we can also look at the above scenarios using some basic calculations. This allows us to build a better intuition and reason about the problem without having to write code. Let's start with some assumptions:
|
||||
|
||||
- two nodes exchanging a single message in batch mode
|
||||
- 10% uniformly random uptime for each node
|
||||
- in HA cache case, 100% uptime of a piece of infrastructure C
|
||||
|
|
@ -116,7 +120,7 @@ A <- ack B (10% chance of arrival)
|
|||
|
||||
With a constant backoff, A will send messages at epoch `1, 2, 3, ...`. With exponential backoff and a multiplier of 2, this would be `1, 2, 4, 8, ...`. Let's assume constant backoff for now, as this is what will influence the success rate and thus the bandwidth multiplier.
|
||||
|
||||
There's a difference between *time to receive* and *time to stop sending*. Assuming each send attempt is independent, it takes on average 10 epochs for A's message to arrive with B. Furthermore:
|
||||
There's a difference between _time to receive_ and _time to stop sending_. Assuming each send attempt is independent, it takes on average 10 epochs for A's message to arrive with B. Furthermore:
|
||||
|
||||
1. A will send messages until it receives an ACK.
|
||||
2. B will send ACK if it receives a message.
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Vac - A Rough Overview"
|
||||
title: "Vac - A Rough Overview"
|
||||
name: 'Vac - A Rough Overview'
|
||||
title: 'Vac - A Rough Overview'
|
||||
date: 2019-08-02 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /vac-overview
|
||||
categories: research
|
||||
category: research
|
||||
summary: Vac is a modular peer-to-peer messaging stack, with a focus on secure messaging. Overview of terms, stack and open problems.
|
||||
---
|
||||
|
||||
|
|
@ -14,24 +14,24 @@ Vac is a **modular peer-to-peer messaging stack, with a focus on secure messagin
|
|||
|
||||
## Basic terms
|
||||
|
||||
*messaging stack*. While the initial focus is on [data sync](https://vac.dev/p2p-data-sync-for-mobile), we are concerned with all layers in the stack. That means all the way from underlying transports, p2p overlays and routing, to initial trust establishment and semantics for things like group chat. The ultimate goal is to give application developers the tools they need to provide secure messaging for their users, so they can focus on their domain expertise.
|
||||
_messaging stack_. While the initial focus is on [data sync](https://vac.dev/p2p-data-sync-for-mobile), we are concerned with all layers in the stack. That means all the way from underlying transports, p2p overlays and routing, to initial trust establishment and semantics for things like group chat. The ultimate goal is to give application developers the tools they need to provide secure messaging for their users, so they can focus on their domain expertise.
|
||||
|
||||
*modular*. Unlike many other secure messaging applications, our goal is not to have a tightly coupled set of protocols, nor is it to reinvent the wheel. Instead, we aim to provide options at each layer in the stack, and build on the shoulders of giants, putting a premimum on interoperability. It's similar in philosophy to projects such as [libp2p](https://libp2p.io/) or [Substrate](https://www.parity.io/substrate/) in that regard. Each choice comes with different trade-offs, and these look different for different applications.
|
||||
_modular_. Unlike many other secure messaging applications, our goal is not to have a tightly coupled set of protocols, nor is it to reinvent the wheel. Instead, we aim to provide options at each layer in the stack, and build on the shoulders of giants, putting a premimum on interoperability. It's similar in philosophy to projects such as [libp2p](https://libp2p.io/) or [Substrate](https://www.parity.io/substrate/) in that regard. Each choice comes with different trade-offs, and these look different for different applications.
|
||||
|
||||
*peer-to-peer*. The protocols we work on are pure p2p, and aim to minimize centralization. This too is in opposition to many initiatives in the secure messaging space.
|
||||
_peer-to-peer_. The protocols we work on are pure p2p, and aim to minimize centralization. This too is in opposition to many initiatives in the secure messaging space.
|
||||
|
||||
*messaging*. By messaging we mean messaging in a generalized sense. This includes both human to human communication, as well machine to machine communication. By messaging we also mean something more fundamental than text messages, we also include things like transactions (state channels, etc) under this moniker.
|
||||
_messaging_. By messaging we mean messaging in a generalized sense. This includes both human to human communication, as well machine to machine communication. By messaging we also mean something more fundamental than text messages, we also include things like transactions (state channels, etc) under this moniker.
|
||||
|
||||
*secure messaging*. Outside of traditional notions of secure messaging, such as ensuring end to end encryption, forward secrecy, avoiding MITM-attacks, etc, we are also concerned with two other forms of secure messaging. We call these *private messaging* and *censorship-resistance*. Private messaging means viewing privacy as a security property, with all that entails. Censorship resistance ties into being p2p, but also in terms of allowing for transports and overlays that can't easily be censored by port blocking, traffic analysis, and similar.
|
||||
_secure messaging_. Outside of traditional notions of secure messaging, such as ensuring end to end encryption, forward secrecy, avoiding MITM-attacks, etc, we are also concerned with two other forms of secure messaging. We call these _private messaging_ and _censorship-resistance_. Private messaging means viewing privacy as a security property, with all that entails. Censorship resistance ties into being p2p, but also in terms of allowing for transports and overlays that can't easily be censored by port blocking, traffic analysis, and similar.
|
||||
|
||||
*Vāc*. Is a Vedic goddess of speech. It also hints at being a vaccine.
|
||||
_Vāc_. Is a Vedic goddess of speech. It also hints at being a vaccine.
|
||||
|
||||
## Protocol stack
|
||||
|
||||
What does this stack look like? We take inspiration from [core](https://tools.ietf.org/html/rfc793) [internet architecture](https://www.ietf.org/rfc/rfc1122.txt), existing [survey work](http://cacr.uwaterloo.ca/techreports/2015/cacr2015-02.pdf) and other [efforts](https://code.briarproject.org/briar/briar/wikis/A-Quick-Overview-of-the-Protocol-Stack) that have been done to decompose the problem into orthogonal pieces. Each layer provides their own set of properties and only interact with the layers it is adjacent to. Note that this is a rough sketch.
|
||||
|
||||
| Layer / Protocol | Purpose | Examples |
|
||||
|-------------------|-----------------------------------|----------------------|
|
||||
| ------------------- | --------------------------------- | -------------------- |
|
||||
| Application layer | End user semantics | 1:1 chat, group chat |
|
||||
| Data Sync | Data consistency | MVDS, BSP |
|
||||
| Secure Transport | Confidentiality, PFS, etc | Double Ratchet, MLS |
|
||||
|
|
@ -51,11 +51,13 @@ In terms of specific properties and trade-offs at each layer, we'll go deeper do
|
|||
With all the pieces involved, this is quite an undertaking. Luckily, a lot of pieces are already in place and can be either incorporated as-is or iterated on. In terms of medium and long term, here's a rough sketch of priorities and open problems.
|
||||
|
||||
1. **Better data sync.** While the current [MVDS](https://specs.vac.dev/specs/mvds.html) works, it is lacking in a few areas:
|
||||
|
||||
- Lack of remote log for mostly-offline offline devices
|
||||
- Better scalability for multi-user chat contexts
|
||||
- Better usability in terms of application layer usage and supporting more transports
|
||||
|
||||
2. **Better transport layer support.** Currently MVDS runs primarily over Whisper, which has a few issues:
|
||||
|
||||
- scalability, being able to run with many nodes
|
||||
- spam-resistance, proof of work is a poor mechanism for heterogeneous devices
|
||||
- no incentivized infrastructure, leading to centralized choke points
|
||||
|
|
@ -67,7 +69,6 @@ In addition to these most immediate concerns, there are other open problems. Som
|
|||
4. **Incentivized and spam-resistant messaging.** Reasons to run infrastructure and not relying on altruistic nodes. For spam resistance, in p2p multicast spam is a big attack vector due to amplification. There are a few interesting directions here, such as EigenTrust, proof of burn with micropayments, and leveraging zero-knowledge proofs.
|
||||
|
||||
5. **Strong privacy guarantees at transport privacy layer**. More rigorous privacy guarantees and explicit trade-offs for metadata protection. Includes Mixnet.
|
||||
|
||||
6. **Censorship-resistant and robust P2P overlay**. NAT traversal; running in the browser; mesh networks; pluggable transports for traffic obfuscation.
|
||||
|
||||
7. **Scalable and decentralized secure conversational security.** Strong security guarantees such as forward secrecy, post compromise security, for large group chats. Includes projects such MLS and extending Double Ratchet.
|
||||
|
|
|
|||
|
|
@ -1,21 +1,21 @@
|
|||
---
|
||||
layout: post
|
||||
name: "P2P Data Sync with a Remote Log"
|
||||
title: "P2P Data Sync with a Remote Log"
|
||||
name: 'P2P Data Sync with a Remote Log'
|
||||
title: 'P2P Data Sync with a Remote Log'
|
||||
date: 2019-10-04 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /remote-log
|
||||
categories: research
|
||||
category: research
|
||||
summary: A research log. Asynchronous P2P messaging? Remote logs to the rescue!
|
||||
image: /assets/img/remote-log.png
|
||||
---
|
||||
|
||||
A big problem when doing end-to-end data sync between mobile nodes is that most devices are offline most of the time. With a naive approach, you quickly run into issues of 'ping-pong' behavior, where messages have to be constantly retransmitted. We saw some basic calculations of what this bandwidth multiplier looks like in a [previous post](https://vac.dev/p2p-data-sync-for-mobile).
|
||||
|
||||
While you could do some background processing, this is really battery-draining, and on iOS these capabilities are limited. A better approach instead is to loosen the constraint that two nodes need to be online at the same time. How do we do this? There are two main approaches, one is the *store and forward model*, and the other is a *remote log*.
|
||||
While you could do some background processing, this is really battery-draining, and on iOS these capabilities are limited. A better approach instead is to loosen the constraint that two nodes need to be online at the same time. How do we do this? There are two main approaches, one is the _store and forward model_, and the other is a _remote log_.
|
||||
|
||||
In the *store and forward* model, we use an intermediate node that forward messages on behalf of the recipient. In the *remote log* model, you instead replicate the data onto some decentralized storage, and have a mutable reference to the latest state, similar to DNS. While both work, the latter is somewhat more elegant and "pure", as it has less strict requirements of an individual node's uptime. Both act as a highly-available cache to smoothen over non-overlapping connection windows between endpoints.
|
||||
In the _store and forward_ model, we use an intermediate node that forward messages on behalf of the recipient. In the _remote log_ model, you instead replicate the data onto some decentralized storage, and have a mutable reference to the latest state, similar to DNS. While both work, the latter is somewhat more elegant and "pure", as it has less strict requirements of an individual node's uptime. Both act as a highly-available cache to smoothen over non-overlapping connection windows between endpoints.
|
||||
|
||||
In this post we are going to describe how such a remote log schema could work. Specifically, how it enhances p2p data sync and takes care of the [following requirements](https://vac.dev/p2p-data-sync-for-mobile):
|
||||
|
||||
|
|
@ -31,12 +31,12 @@ In this post we are going to describe how such a remote log schema could work. S
|
|||
|
||||
A remote log is a replication of a local log. This means a node can read data from a node that is offline.
|
||||
|
||||
The spec is in an early draft stage and can be found [here](https://github.com/vacp2p/specs/pull/16). A very basic [spike](https://en.wikipedia.org/wiki/Spike_(software_development)) / proof-of-concept can be found [here](https://github.com/vacp2p/research/tree/master/remote_log).
|
||||
The spec is in an early draft stage and can be found [here](https://github.com/vacp2p/specs/pull/16). A very basic [spike](<https://en.wikipedia.org/wiki/Spike_(software_development)>) / proof-of-concept can be found [here](https://github.com/vacp2p/research/tree/master/remote_log).
|
||||
|
||||
### Definitions
|
||||
|
||||
| Term | Definition |
|
||||
| ----------- | -------------------------------------------------------------------------------------- |
|
||||
| ---------- | ------------------------------------------------------------------------- |
|
||||
| CAS | Content-addressed storage. Stores data that can be addressed by its hash. |
|
||||
| NS | Name system. Associates mutable data to a name. |
|
||||
| Remote log | Replication of a local log at a different location. |
|
||||
|
|
@ -47,10 +47,10 @@ There are four fundamental roles:
|
|||
|
||||
1. Alice
|
||||
2. Bob
|
||||
2. Name system (NS)
|
||||
3. Content-addressed storage (CAS)
|
||||
3. Name system (NS)
|
||||
4. Content-addressed storage (CAS)
|
||||
|
||||
The *remote log* is the data format of what is stored in the name system.
|
||||
The _remote log_ is the data format of what is stored in the name system.
|
||||
|
||||
"Bob" can represent anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS.
|
||||
|
||||
|
|
@ -75,22 +75,22 @@ The remote log lets receiving nodes know what data they are missing. Depending o
|
|||
A remote log is simply a mapping from message identifiers to their corresponding address in a CAS:
|
||||
|
||||
| Message Identifier (H1) | CAS Hash (H2) |
|
||||
| ---------------- |---------------|
|
||||
| ----------------------- | ------------- |
|
||||
| H1_3 | H2_3 |
|
||||
| H1_2 | H2_2 |
|
||||
| H1_1 | H2_1 |
|
||||
| | |
|
||||
| *address to next page* |
|
||||
| _address to next page_ |
|
||||
|
||||
The numbers here corresponds to messages. Optionally, the content itself can be included, just like it normally would be sent over the wire. This bypasses the need for a dedicated CAS and additional round-trips, with a trade-off in bandwidth usage.
|
||||
|
||||
| Message Identifier (H1) | Content |
|
||||
| ---------------- |---------------|
|
||||
| ----------------------- | ------- |
|
||||
| H1_3 | C3 |
|
||||
| H1_2 | C2 |
|
||||
| H1_1 | C1 |
|
||||
| | |
|
||||
| *address to next page* |
|
||||
| _address to next page_ |
|
||||
|
||||
Both patterns can be used in parallel, e,g. by storing the last `k` messages directly and use CAS pointers for the rest. Together with the `next_page` page semantics, this gives users flexibility in terms of bandwidth and latency/indirection, all the way from a simple linked list to a fully replicated log. The latter is useful for things like backups on durable storage.
|
||||
|
||||
|
|
@ -103,13 +103,13 @@ Both patterns can be used in parallel, e,g. by storing the last `k` messages dir
|
|||
The spec is still in an early draft stage, so it is expected to change. Same with the proof of concept. More work is needed on getting a fully featured proof of concept with specific CAS and NAS instances. E.g. Swarm and Swarm Feeds, or IPFS and IPNS, or something else.
|
||||
|
||||
For data sync in general:
|
||||
|
||||
- Make consistency guarantees more explicit for app developers with support for sequence numbers and DAGs, as well as the ability to send non-synced messages. E.g. ephemeral typing notifications, linear/sequential history and casual consistency/DAG history
|
||||
- Better semantics and scalability for multi-user sync contexts, e.g. CRDTs and joining multiple logs together
|
||||
- Better usability in terms of application layer usage (data sync clients) and supporting more transports
|
||||
|
||||
---
|
||||
|
||||
|
||||
PS1. Thanks everyone who submitted great [logo proposals](https://explorer.bounties.network/bounty/3389) for Vac!
|
||||
|
||||
PPS2. Next week on October 10th decanus and I will be presenting Vac at [Devcon](https://devcon.org/agenda), come say hi :)
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Feasibility Study: Semaphore rate limiting through zkSNARKs"
|
||||
title: "Feasibility Study: Semaphore rate limiting through zkSNARKs"
|
||||
name: 'Feasibility Study: Semaphore rate limiting through zkSNARKs'
|
||||
title: 'Feasibility Study: Semaphore rate limiting through zkSNARKs'
|
||||
date: 2019-11-08 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /feasibility-semaphore-rate-limiting-zksnarks
|
||||
categories: research
|
||||
category: research
|
||||
summary: A research log. Zero knowledge signaling as a rate limiting mechanism to prevent spam in p2p networks.
|
||||
image: /assets/img/peacock-signaling.jpg
|
||||
discuss: https://forum.vac.dev/t/discussion-feasibility-study-semaphore-rate-limiting-through-zksnarks/21
|
||||
|
|
@ -33,13 +33,13 @@ In addition to above, there are a lot of related problems that share similaritie
|
|||
|
||||
## Basic terminology
|
||||
|
||||
A *zero-knowledge proof* allows a *prover* to show a *verifier* that they know something, without revealing what that something is. This means you can do trust-minimized computation that is also privacy preserving. As a basic example, instead of showing your ID when going to a bar you simply give them a proof that you are over 18, without showing the doorman your id.
|
||||
A _zero-knowledge proof_ allows a _prover_ to show a _verifier_ that they know something, without revealing what that something is. This means you can do trust-minimized computation that is also privacy preserving. As a basic example, instead of showing your ID when going to a bar you simply give them a proof that you are over 18, without showing the doorman your id.
|
||||
|
||||
*zkSNARKs* is a form of zero-knowledge proofs. There are many types of zero-knowledge proofs, and the field is evolving rapidly. They come with various trade-offs in terms of things such as: trusted setup, cryptographic assumptions, proof/verification key size, proof/verification time, proof size, etc. See section below for more.
|
||||
_zkSNARKs_ is a form of zero-knowledge proofs. There are many types of zero-knowledge proofs, and the field is evolving rapidly. They come with various trade-offs in terms of things such as: trusted setup, cryptographic assumptions, proof/verification key size, proof/verification time, proof size, etc. See section below for more.
|
||||
|
||||
*Semaphore* is a framework/library/construct on top of zkSNARks. It allows for zero-knowledge signaling, specifically on top of Ethereum. This means an approved user can broadcast some arbitrary string without revealing their identity, given some specific constraints. An approved user is someone who has been added to a certain merkle tree. See [current Github home](https://github.com/kobigurk/semaphore) for more.
|
||||
_Semaphore_ is a framework/library/construct on top of zkSNARks. It allows for zero-knowledge signaling, specifically on top of Ethereum. This means an approved user can broadcast some arbitrary string without revealing their identity, given some specific constraints. An approved user is someone who has been added to a certain merkle tree. See [current Github home](https://github.com/kobigurk/semaphore) for more.
|
||||
|
||||
*Circom* is a DSL for writing arithmetic circuits that can be used in zkSNARKs, similar to how you might write a NAND gate. See [Github](https://github.com/iden3/circom) for more.
|
||||
_Circom_ is a DSL for writing arithmetic circuits that can be used in zkSNARKs, similar to how you might write a NAND gate. See [Github](https://github.com/iden3/circom) for more.
|
||||
|
||||
## Basic flow
|
||||
|
||||
|
|
@ -47,24 +47,19 @@ We start with a private voting example, and then extend it to the slashable rate
|
|||
|
||||
1. A user registers an identity (arbitrary keypair), along with a small fee, to a smart contract. This adds them to a merkle tree and allows them to prove that they are member of that group, without revealing who they are.
|
||||
|
||||
2. When a user wants to send a message, they compute a zero-knowledge proof. This ensures certain invariants, have some *public outputs*, and can be verified by anyone (including a smart contract).
|
||||
|
||||
2. When a user wants to send a message, they compute a zero-knowledge proof. This ensures certain invariants, have some _public outputs_, and can be verified by anyone (including a smart contract).
|
||||
3. Any node can verify the proof, including smart contracts on chain (as of Byzantinum HF). Additionally, a node can have rules for the public output. In the case of voting, one such rule is that a specific output hash has to be equal to some predefined value, such as "2020-01-01 vote on Foo Bar for president".
|
||||
|
||||
4. Because of how the proof is constructed, and the rules around output values, this ensures that: a user is part of the approved set of voters and that a user can only vote once.
|
||||
|
||||
5. As a consequence of above, we have a system where registered users can only vote once, no one can see who voted for what, and this can all be proven and verified.
|
||||
|
||||
### Rate limiting example
|
||||
|
||||
In the case of rate limiting, we do want nodes to send multiple messages. This changes step 3-5 above somewhat.
|
||||
|
||||
*NOTE: It is a bit more involved than this, and if we precompute proofs the flow might look a bit different. But the general idea is the same*.
|
||||
_NOTE: It is a bit more involved than this, and if we precompute proofs the flow might look a bit different. But the general idea is the same_.
|
||||
|
||||
1. Instead of having a rule that you can only vote once, we have a rule that you can only send a message per epoch. Epoch here can be every second, as defined by UTC date time +-20s.
|
||||
|
||||
2. Additionally, if a users sends more than one message per epoch, one of the public outputs is a random share of a private key. Using Shamir's Secret Sharing (similar to a multisig) and 2/3 key share as an example threshold: in the normal case only 1/3 private keys is revealed, which is insufficient to have access. In the case where two messages are sent in an epoch, probabilistically 2/3 shares is sufficient to have access to the key (unless you get the same random share of the key).
|
||||
|
||||
3. This means any untrusted user who detects a spamming user, can use it to access their private key corresponding to funds in the contract, and thus slash them.
|
||||
|
||||
4. As a consequence of above, we have a system where registered users can only messages X times per epoch, and no one can see who is sending what messages. Additionally, if a user is violating the above rate limit, they can be punished and any user can profit from it.
|
||||
|
|
@ -149,4 +144,4 @@ For example, this might also include leveraging largely ready made solutions suc
|
|||
|
||||
Thanks to Barry Whitehat for patient explanation and pointers. Thanks to WJ for helping with runtime issues.
|
||||
|
||||
*Peacock header image from [Tonos](https://en.wikipedia.org/wiki/File:Flickr_-_lo.tangelini_-_Tonos_(1).jpg).*
|
||||
_Peacock header image from [Tonos](<https://en.wikipedia.org/wiki/File:Flickr_-_lo.tangelini_-_Tonos_(1).jpg>)._
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Fixing Whisper with Waku"
|
||||
title: "Fixing Whisper with Waku"
|
||||
name: 'Fixing Whisper with Waku'
|
||||
title: 'Fixing Whisper with Waku'
|
||||
date: 2019-12-03 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /fixing-whisper-with-waku
|
||||
categories: research
|
||||
category: research
|
||||
summary: A research log. Why Whisper doesn't scale and how to fix it.
|
||||
image: /assets/img/whisper_scalability.png
|
||||
discuss: https://forum.vac.dev/t/discussion-fixing-whisper-with-waku/27
|
||||
|
|
@ -41,7 +41,7 @@ In this post, we'll focus on the first problem, which is scalability through ban
|
|||
|
||||
## Whisper theoretical scalability model
|
||||
|
||||
*(Feel free to skip this section if you want to get right to the results).*
|
||||
_(Feel free to skip this section if you want to get right to the results)._
|
||||
|
||||
There's widespread implicit knowledge that Whisper "doesn't scale", but it is less understood exactly why. This theoretical model attempts to encode some characteristics of it. Specifically for use case such as one by Status (see [Status Whisper usage
|
||||
spec](https://github.com/status-im/specs/blob/master/status-whisper-usage-spec.md)).
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Waku Update"
|
||||
title: "Waku Update"
|
||||
name: 'Waku Update'
|
||||
title: 'Waku Update'
|
||||
date: 2020-02-14 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /waku-update
|
||||
categories: research
|
||||
category: research
|
||||
summary: A research log. What's the current state of Waku? How many users does it support? What are the bottlenecks? What's next?
|
||||
image: /assets/img/waku_infrastructure_sky.jpg
|
||||
discuss: https://forum.vac.dev/t/waku-update-where-are-we-at/34
|
||||
|
|
@ -51,12 +51,13 @@ We have a [simulation](https://github.com/status-im/nimbus/tree/master/waku#test
|
|||
This is our current understanding of how many users a network running Waku can support. Specifically in the context of the Status chat app, since that's the most immediate consumer of Waku. It should generalize fairly well to most deployments.
|
||||
|
||||
**tl;dr (for Status app):**
|
||||
|
||||
- beta: 100 DAU
|
||||
- v1: 1k DAU
|
||||
- v1.1 (waku only): 10k DAU (up to x10 with deployment hotfixes)
|
||||
- v1.2 (waku+dns): 100k DAU (can optionally be folded into v1.1)
|
||||
|
||||
*Assuming 10 concurrent users = 100 DAU. Estimate uncertainty increases for each order of magnitude until real-world data is observed.*
|
||||
_Assuming 10 concurrent users = 100 DAU. Estimate uncertainty increases for each order of magnitude until real-world data is observed._
|
||||
|
||||
As far as we know right now, these are the bottlenecks we have:
|
||||
|
||||
|
|
@ -83,7 +84,7 @@ For light node, bloom filter is set to almost 10% false positive (bloom filter:
|
|||
**Star network:**
|
||||
|
||||
| Description | Peers | Valid | Invalid |
|
||||
|-----------------|-------|-------|---------|
|
||||
| --------------- | ----- | ----- | ------- |
|
||||
| Master node | 7 | 10001 | 0 |
|
||||
| Full node 1 | 3 | 10001 | 0 |
|
||||
| Full node 2 | 1 | 10001 | 0 |
|
||||
|
|
@ -96,7 +97,7 @@ For light node, bloom filter is set to almost 10% false positive (bloom filter:
|
|||
**Full mesh:**
|
||||
|
||||
| Description | Peers | Valid | Invalid |
|
||||
|-----------------|-------|-------|---------|
|
||||
| --------------- | ----- | ----- | ------- |
|
||||
| Full node 0 | 7 | 10001 | 20676 |
|
||||
| Full node 1 | 7 | 10001 | 9554 |
|
||||
| Full node 2 | 5 | 10001 | 23304 |
|
||||
|
|
@ -107,6 +108,7 @@ For light node, bloom filter is set to almost 10% false positive (bloom filter:
|
|||
| Waku light node | 2 | 1 | 1 |
|
||||
|
||||
Things to note:
|
||||
|
||||
- Whisper light node with ~10% false positive gets ~10% of total traffic
|
||||
- Waku light node gets ~1000x less envelopes than Whisper light node
|
||||
- Full mesh results in a lot more duplicate messages, expect for Waku light node
|
||||
|
|
@ -141,4 +143,4 @@ For the third bottleneck, a likely candidate for fixing this is Kademlia routing
|
|||
|
||||
## Acknowledgements
|
||||
|
||||
*Image from "caged sky" by mh.xbhd.org is licensed under CC BY 2.0 (https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e)*
|
||||
_Image from "caged sky" by mh.xbhd.org is licensed under CC BY 2.0 (https://ccsearch.creativecommons.org/photos/a9168311-78de-4cb7-a6ad-f92be8361d0e)_
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "DNS Based Discovery"
|
||||
title: "DNS Based Discovery"
|
||||
name: 'DNS Based Discovery'
|
||||
title: 'DNS Based Discovery'
|
||||
date: 2020-02-7 12:00:00 +0100
|
||||
author: dean
|
||||
published: true
|
||||
permalink: /dns-based-discovery
|
||||
categories: research
|
||||
category: research
|
||||
summary: A look at EIP-1459 and the benefits of DNS based discovery.
|
||||
---
|
||||
|
||||
|
|
@ -20,7 +20,7 @@ How do we do this?
|
|||
|
||||
[EIP 1459: Node Discovery via DNS](https://eips.ethereum.org/EIPS/eip-1459), which is one of the strategies we are using for discovering waku nodes. [EIP-1459](https://eips.ethereum.org/EIPS/eip-1459) is a DNS-based discovery protocol that stores [merkle trees](https://en.wikipedia.org/wiki/Merkle_tree) in DNS records which contain connection information for nodes.
|
||||
|
||||
*Waku is our fork of Whisper. Oskar recently wrote an [entire post](https://vac.dev/fixing-whisper-with-waku) explaining it. In short, Waku is our method of fixing the shortcomings of Whisper in a more iterative fashion. You can find the specification [here](https://specs.vac.dev/specs/waku/waku.html)*
|
||||
_Waku is our fork of Whisper. Oskar recently wrote an [entire post](https://vac.dev/fixing-whisper-with-waku) explaining it. In short, Waku is our method of fixing the shortcomings of Whisper in a more iterative fashion. You can find the specification [here](https://specs.vac.dev/specs/waku/waku.html)_
|
||||
|
||||
DNS-based methods for bootstrapping p2p networks are quite popular. Even Bitcoin uses it, but it uses a concept called DNS seeds, which are just DNS servers that are configured to return a list of randomly selected nodes from the network upon being queried. This means that although these seeds are hardcoded in the client, the IP addresses of actual nodes do not have to be.
|
||||
|
||||
|
|
@ -64,6 +64,7 @@ There are multiple reasons, but the main one is **security**. In the Bitcoin exa
|
|||
Let's dive into this. Firstly, a client that is using these DNS records for discovery must know the public key corresponding to the private key controlled by the entity creating the list. This is because the entire list is signed using a secp256k1 private key, giving the client the ability to authenticate the list and know that it has not been tampered with by some external party.
|
||||
|
||||
So that already makes this a lot safer than the method Bitcoin uses. But how are these lists even stored? As previously stated they are stored using **merkle trees** as follows:
|
||||
|
||||
- The root of the tree is stored in a [`TXT` record](https://simpledns.plus/help/txt-records), this record contains the tree's root hash, a sequence number which is incremented every time the tree is updated and a signature as stated above.
|
||||
|
||||
Additionally, there is also a root hash to a second tree called a **link tree**, it contains the information to different lists. This link tree allows us to delegate trust and build a graph of multiple merkle trees stored across multiple DNS names.
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "What Would a WeChat Replacement Need?"
|
||||
title: "What Would a WeChat Replacement Need?"
|
||||
name: 'What Would a WeChat Replacement Need?'
|
||||
title: 'What Would a WeChat Replacement Need?'
|
||||
date: 2020-04-16 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /wechat-replacement-need
|
||||
categories: research
|
||||
category: research
|
||||
summary: What would a self-sovereign, private, censorship-resistant and open alternative to WeChat look like?
|
||||
image: /assets/img/tianstatue.jpg
|
||||
discuss: https://forum.vac.dev/t/discussion-what-would-a-wechat-replacement-need/42
|
||||
|
|
@ -84,11 +84,12 @@ We want something that is self-sovereign, private, censorship-resistant and open
|
|||
**Open source**. Related to above, where we must be able to inspect the software to know that it functions as advertised and hasn't been compromised, e.g. by uploading private data to a third party.
|
||||
|
||||
Some of these are graded and a bit subtle, i.e.:
|
||||
|
||||
- Censorship resistance would ideally be able to absorb Internet shutdowns. This would require an extensive MANET/meshnet infrastructure, which while desirable, requires a lot of challenges to be overcome to be feasible.
|
||||
- Privacy would ideally make all actions (optionally) totally anoymous, though this may incur undue costs on bandwidth and latency, which impacts user experience.
|
||||
- Decentralization, certain topologies, such as DHTs, are efficient and quite decentralized but still have some centralized aspects, which makes it attackable in various ways. Ditto for blockchains compared with bearer instruments which requires some coordinating infrastructure, compared with naturally occuring assets such as precious metals.
|
||||
- "Discover people" and striving for "total anonymity" might initially seem incompatible. The idea is to provide for sane defaults, and then allow people to decide how much information they want to disclose. This is the essence of privacy.
|
||||
- Users often want *some* form of moderation to get a good user experience, which can be seen as a form of censorship. The idea to raise the bar on the basics, the fundamental infrastructure. If individuals or specific communities want certain moderation mechanisms, that is still a compatible requirement.
|
||||
- Users often want _some_ form of moderation to get a good user experience, which can be seen as a form of censorship. The idea to raise the bar on the basics, the fundamental infrastructure. If individuals or specific communities want certain moderation mechanisms, that is still a compatible requirement.
|
||||
|
||||
### Counterpoint 1
|
||||
|
||||
|
|
@ -115,9 +116,10 @@ Starting from the most basic: what is an account and how do you get one? With mo
|
|||
Now, we could argue these app providers can do what they want. And they are right, in a very narrow sense. As apps like WeChat (and Google) become general-purpose platforms, they become more and more ingrained in our everyday lives. They start to provide utilities that we absolutely require to work to go about our day, such as paying for food or transportation. This means we need higher standard than this.
|
||||
|
||||
Justifications for requiring phone numbers are usually centered around three claims:
|
||||
1) Avoiding spam
|
||||
2) Tying your account to your real name, for various reasons
|
||||
3) Using as a commonly shared identifier as a social network discovery mechanism
|
||||
|
||||
1. Avoiding spam
|
||||
2. Tying your account to your real name, for various reasons
|
||||
3. Using as a commonly shared identifier as a social network discovery mechanism
|
||||
|
||||
Of course, many services require more than phone numbers. E.g. email, other forms of personal data such as voice recording, linking a bank account, and so on.
|
||||
|
||||
|
|
@ -246,19 +248,11 @@ Corey, Dean, Jacek.
|
|||
## References
|
||||
|
||||
[^1]: Mandatory SIM card registration laws: https://privacyinternational.org/long-read/3018/timeline-sim-card-registration-laws
|
||||
|
||||
[^2]: On WeChat keyword censorship: https://citizenlab.ca/2016/11/wechat-china-censorship-one-app-two-systems/
|
||||
|
||||
[^3]: Net Neutrality: https://www.eff.org/issues/net-neutrality
|
||||
|
||||
[^4]: ISP centralization: https://ilsr.org/repealing-net-neutrality-puts-177-million-americans-at-risk/
|
||||
|
||||
[^5]: Incentives Build Robustness in BitTorrent bittorrent.org/bittorrentecon.pdf
|
||||
|
||||
[^6]: The Case for Electronic Cash: https://coincenter.org/files/2019-02/the-case-for-electronic-cash-coin-center.pdf
|
||||
|
||||
[^7]: Money, blockchains, and social scalability: http://unenumerated.blogspot.com/2017/02/money-blockchains-and-social-scalability.html
|
||||
|
||||
[^8]: Zcash private transactions (partial paywall): https://www.theblockcrypto.com/genesis/48413/an-analysis-of-zcashs-private-transactions
|
||||
|
||||
[^9]: Shielded transactions usage (stats page 404s): https://z.cash/support/faq/
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Feasibility Study: Discv5"
|
||||
title: "Feasibility Study: Discv5"
|
||||
name: 'Feasibility Study: Discv5'
|
||||
title: 'Feasibility Study: Discv5'
|
||||
date: 2020-04-27 12:00:00 +0800
|
||||
author: dean
|
||||
published: true
|
||||
permalink: /feasibility-discv5
|
||||
categories: research
|
||||
category: research
|
||||
summary: Looking at discv5 and the theoretical numbers behind finding peers.
|
||||
discuss: https://discuss.status.im/t/discv5-feasibility-study/1632
|
||||
---
|
||||
|
|
@ -38,6 +38,7 @@ In order to answer our questions, we formulated the problem with a simple method
|
|||
## Overview
|
||||
|
||||
The main things we wanted to investigate was the overhead on finding a peer. This means we wanted to look at both the bandwidth, latency and effectiveness of this. There are 2 methods which we can use to find a peer:
|
||||
|
||||
- We can find a peer with a specific ID, using normal lookup methods as documented by Kademlia.
|
||||
- We can find a peer that advertises a capability, this is possible using either capabilities advertised in the ENR or through [topic tables](https://github.com/ethereum/devp2p/blob/master/discv5/discv5-theory.md#topic-advertisement).
|
||||
|
||||
|
|
@ -76,6 +77,7 @@ Research has shown that finding a node in the DHT has a relatively low effect on
|
|||
One of the main blockers right now is figuring out what the CPU and memory usage of discv5 is on mobile phones, this is a large blocker as it affects one of the core problems for us. We need to consider whether discv5 is an upgrade as it allows us to simplify our current discovery process or if it is too much of an overhead for resource restricted devices. The topic table feature could largely enhance discovery however it is not yet implemented. Given that CPU and memory isn't too high, discv5 could probably be used as the other issues are more "features" than large scale issues. Implementing it would already reduce the ability for state level adversaries to censor our nodes.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- Oskar Thoren
|
||||
- Dmitry Shmatko
|
||||
- Kim De Mey
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "From Kademlia to Discv5"
|
||||
title: "From Kademlia to Discv5"
|
||||
name: 'From Kademlia to Discv5'
|
||||
title: 'From Kademlia to Discv5'
|
||||
date: 2020-04-9 16:00:00 +0100
|
||||
author: dean
|
||||
published: true
|
||||
permalink: /kademlia-to-discv5
|
||||
categories: research
|
||||
category: research
|
||||
summary: A quick history of discovery in peer-to-peer networks, along with a look into discv4 and discv5, detailing what they are, how they work and where they differ.
|
||||
---
|
||||
|
||||
|
|
@ -48,6 +48,7 @@ XOR 10011001
|
|||
The top in decimal numbers means that the distance between `153` and `50` is `171`.
|
||||
|
||||
There are several reasons why `XOR` was taken:
|
||||
|
||||
1. The distance from one ID to itself will be `0`.
|
||||
2. Distance is symmetric, A to B is the same as B to A.
|
||||
3. Follows triangle inequality, if `A`, `B` and `C` are points on a triangle then the distance `A` to `B` is closer or equal to that of `A` to `C` plus the one from `B` to `C`.
|
||||
|
|
@ -67,7 +68,7 @@ For Kademlia nodes to support these functions, there are several messages with w
|
|||
- `FINDNODE` - Returns the closest nodes requested to a given ID.
|
||||
- `FINDVALUE` - The same as `FINDNODE`, except if a node stores the specific value it will return it directly.
|
||||
|
||||
*This is a **very** simplified explanation of Kademlia and skips various important details. For the full description, make sure to check out the [paper](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf) or a more in-depth [design specification](http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html)*
|
||||
_This is a **very** simplified explanation of Kademlia and skips various important details. For the full description, make sure to check out the [paper](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf) or a more in-depth [design specification](http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html)_
|
||||
|
||||
## Discv4
|
||||
|
||||
|
|
@ -81,7 +82,7 @@ Additionally, discv4 adds mutual endpoint verification. This is meant to ensure
|
|||
|
||||
Finally, all discv4 nodes are expected to maintain up-to-date ENR records. These contain information about a node. They can be requested from any node using a discv4-specific packet called `ENRRequest`.
|
||||
|
||||
*If you want some more details on ENRs, check out one of my posts ["Network Addresses in Ethereum"](https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/)*
|
||||
_If you want some more details on ENRs, check out one of my posts ["Network Addresses in Ethereum"](https://dean.eigenmann.me/blog/2020/01/21/network-addresses-in-ethereum/)_
|
||||
|
||||
Discv4 comes with its own range of problems however. Let's look at a few of them.
|
||||
|
||||
|
|
@ -103,7 +104,7 @@ Logarithmic distance means we first calculate the distance and then run it throu
|
|||
log2(A xor B)
|
||||
```
|
||||
|
||||
And the second, more important change, is that discv5 aims at solving one of the biggest issues of discv4: the differentiation of sub-protocols. It does this by adding topic tables. Topic tables are [first in first out](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) lists that contain nodes which have advertised that they provide a specific service. Nodes get themselves added to this list by registering `ads` on their peers.
|
||||
And the second, more important change, is that discv5 aims at solving one of the biggest issues of discv4: the differentiation of sub-protocols. It does this by adding topic tables. Topic tables are [first in first out](<https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)>) lists that contain nodes which have advertised that they provide a specific service. Nodes get themselves added to this list by registering `ads` on their peers.
|
||||
|
||||
As of writing, there is still an issue with this proposal. There is currently no efficient way for a node to place `ads` on multiple peers, since it would require separate requests for every peer which is inefficient in a large-scale network.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Problem 4 Wallet stacking sats block height"
|
||||
title: "Problem 4 Wallet stacking sats block height"
|
||||
name: 'Problem 4 Wallet stacking sats block height'
|
||||
title: 'Problem 4 Wallet stacking sats block height'
|
||||
date: 2020-07-01 12:00:00 +0800
|
||||
author: franck
|
||||
published: true
|
||||
permalink: /status-ipsum-4
|
||||
category: problem
|
||||
summary: "Status ipsum lorem exequoe"
|
||||
summary: 'Status ipsum lorem exequoe'
|
||||
image: /assets/img/js-waku-gist.png
|
||||
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
|
||||
---
|
||||
|
|
@ -117,7 +117,7 @@ we were able to fine tune the API to provide a simple and succinct interface.
|
|||
You can start a node, connect to other nodes and send a message in less than ten lines of code:
|
||||
|
||||
```javascript
|
||||
import { Waku } from "js-waku";
|
||||
import { Waku } from 'js-waku';
|
||||
|
||||
const waku = await Waku.create({});
|
||||
|
||||
|
|
@ -125,8 +125,8 @@ const nodes = await getStatusFleetNodes();
|
|||
await Promise.all(nodes.map((addr) => waku.dial(addr)));
|
||||
|
||||
const msg = WakuMessage.fromUtf8String(
|
||||
"Here is a message!",
|
||||
"/my-cool-app/1/my-use-case/proto"
|
||||
'Here is a message!',
|
||||
'/my-cool-app/1/my-use-case/proto'
|
||||
);
|
||||
await waku.relay.send(msg);
|
||||
```
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ date: 2020-07-01 12:00:00 +0800
|
|||
author: oskarth
|
||||
published: true
|
||||
permalink: /waku-v2-plan
|
||||
categories: research
|
||||
category: research
|
||||
summary: Read about our plans for Waku v2, moving to libp2p, better routing, adaptive nodes and accounting!
|
||||
image: /assets/img/status_scaling_model_fig4.png
|
||||
discuss: https://forum.vac.dev/t/waku-version-2-pitch/52
|
||||
|
|
@ -14,7 +14,7 @@ discuss: https://forum.vac.dev/t/waku-version-2-pitch/52
|
|||
|
||||
**tldr: The Waku network is fragile and doesn't scale. Here's how to solve it.**
|
||||
|
||||
*NOTE: This post was originally written with Status as a primary use case in mind, which reflects how we talk about some problems here. However, Waku v2 is a general-purpose private p2p messaging protocol, especially for people running in resource restricted environments.*
|
||||
_NOTE: This post was originally written with Status as a primary use case in mind, which reflects how we talk about some problems here. However, Waku v2 is a general-purpose private p2p messaging protocol, especially for people running in resource restricted environments._
|
||||
|
||||
# Problem
|
||||
|
||||
|
|
@ -26,12 +26,12 @@ Based on user acquisition models, the initial goal is to support 100k DAU in Sep
|
|||
|
||||
With the Status Scaling Model we have studied the current bottlenecks as a function of concurrent users (CCU) and daily active users (DAU). Here are the conclusions.
|
||||
|
||||
****1. Connection limits****. With 100 full nodes we reach ~10k CCU based on connection limits. This can primarily be addressed by increasing the number of nodes (cluster or user operated). This assumes node discovery works. It is also worth investigating the limitations of max number of connections, though this is likely to be less relevant for user-operated nodes. For a user-operated network, this means 1% of users have to run a full node. See Fig 1-2.
|
||||
\***\*1. Connection limits\*\***. With 100 full nodes we reach ~10k CCU based on connection limits. This can primarily be addressed by increasing the number of nodes (cluster or user operated). This assumes node discovery works. It is also worth investigating the limitations of max number of connections, though this is likely to be less relevant for user-operated nodes. For a user-operated network, this means 1% of users have to run a full node. See Fig 1-2.
|
||||
|
||||
****2. Bandwidth as a bottleneck****. We notice that memory usage appears to not be
|
||||
\***\*2. Bandwidth as a bottleneck\*\***. We notice that memory usage appears to not be
|
||||
the primary bottleneck for full nodes, and the bottleneck is still bandwidth. To support 10k DAU, and full nodes with an amplification factor of 25 the required Internet speed is ~50 Mbps, which is a fast home Internet connection. For ~100k DAU only cloud-operated nodes can keep up (500 Mbps). See Fig 3-5.
|
||||
|
||||
****3. Amplification factors****. Reducing amplification factors with better routing, would have a high impact, but it is likely we'd need additional measures as well, such as topic sharding or similar. See Fig 8-13.
|
||||
\***\*3. Amplification factors\*\***. Reducing amplification factors with better routing, would have a high impact, but it is likely we'd need additional measures as well, such as topic sharding or similar. See Fig 8-13.
|
||||
|
||||
Figure 1-5:
|
||||
|
||||
|
|
@ -79,7 +79,6 @@ Let's first look at the baseline, and then go into some of the tracks and their
|
|||
|
||||
Here's where we are at now. In reality, the amplification factor are likely even worse than this (15 in the graph below), up to 20-30. Especially with an open network, where we can't easily control connectivity and availability of nodes. Left unchecked, with a full mesh, it could even go as high x100, though this is likely excessive and can be dialed down. See scaling model for more details.
|
||||
|
||||
|
||||
|
||||
|
||||
## Track 1 - Move to libp2p
|
||||
|
|
@ -100,7 +99,6 @@ Why can't we use Waku topics for routing directly? PubSub over libp2p isn't buil
|
|||
|
||||
|
||||
|
||||
|
||||
Moving to FloodSub over libp2p would also be an opportunity to clean up and simplify some components that are no longer needed in the Waku v1 protocol, see point below.
|
||||
|
||||
Very experimental and incomplete libp2p support can be found in the nim-waku repo under v2: <https://github.com/status-im/nim-waku>
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Problem No. 2, very long title Bitcoin ipsum dolor sit amet. Peer-to-peer whitepaper SHA-256 bitcoin."
|
||||
title: "Problem No. 2, very long title Bitcoin ipsum dolor sit amet. Peer-to-peer whitepaper SHA-256 bitcoin."
|
||||
name: 'Problem No. 2, very long title Bitcoin ipsum dolor sit amet. Peer-to-peer whitepaper SHA-256 bitcoin.'
|
||||
title: 'Problem No. 2, very long title Bitcoin ipsum dolor sit amet. Peer-to-peer whitepaper SHA-256 bitcoin.'
|
||||
date: 2020-09-28 12:00:00 +0800
|
||||
author: franck
|
||||
published: true
|
||||
permalink: /status-ipsum-2
|
||||
category: problem
|
||||
summary: "Status ipsum lorem exequoe"
|
||||
summary: 'Status ipsum lorem exequoe'
|
||||
image: /assets/img/js-waku-gist.png
|
||||
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
|
||||
---
|
||||
|
|
@ -117,7 +117,7 @@ we were able to fine tune the API to provide a simple and succinct interface.
|
|||
You can start a node, connect to other nodes and send a message in less than ten lines of code:
|
||||
|
||||
```javascript
|
||||
import { Waku } from "js-waku";
|
||||
import { Waku } from 'js-waku';
|
||||
|
||||
const waku = await Waku.create({});
|
||||
|
||||
|
|
@ -125,8 +125,8 @@ const nodes = await getStatusFleetNodes();
|
|||
await Promise.all(nodes.map((addr) => waku.dial(addr)));
|
||||
|
||||
const msg = WakuMessage.fromUtf8String(
|
||||
"Here is a message!",
|
||||
"/my-cool-app/1/my-use-case/proto"
|
||||
'Here is a message!',
|
||||
'/my-cool-app/1/my-use-case/proto'
|
||||
);
|
||||
await waku.relay.send(msg);
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Waku v2 Update"
|
||||
title: "Waku v2 Update"
|
||||
name: 'Waku v2 Update'
|
||||
title: 'Waku v2 Update'
|
||||
date: 2020-09-28 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /waku-v2-update
|
||||
categories: research
|
||||
category: research
|
||||
summary: A research log. Read on to find out what is going on with Waku v2, a messaging protocol. What has been happening? What is coming up next?
|
||||
image: /assets/img/vac.png
|
||||
discuss: https://forum.vac.dev/t/discussion-waku-v2-update/56
|
||||
|
|
@ -72,16 +72,16 @@ Now that we know what the current state is, what is still missing? what are the
|
|||
|
||||
While we are getting closer to closing out work for track 1, there are still a few things missing from the initial scope:
|
||||
|
||||
1) Store and filter protocols need to be finished. This means basic spec, implementation, API integration and proven to work in a testnet. All of these are work in progress and expected to be done very soon. Once the store protocol is done in a basic form, it needs further improvements to make it production ready, at least on a spec/basic implementation level.
|
||||
1. Store and filter protocols need to be finished. This means basic spec, implementation, API integration and proven to work in a testnet. All of these are work in progress and expected to be done very soon. Once the store protocol is done in a basic form, it needs further improvements to make it production ready, at least on a spec/basic implementation level.
|
||||
|
||||
2) Core integration was mentioned in scope for track 1 initially. This work has stalled a bit, largely due to organizational bandwidth and priorities. While there is a Nim Node API that in theory is ready to be used, having it be used in e.g. Status desktop or mobile app is a different matter. The team responsible for this at Status ([status-nim](github.com/status-im/status-nim)) has been making progress on getting nim-waku v1 integrated, and is expected to look into nim-waku v2 integration soon. One thing that makes this a especially tricky is the difference in interface between Waku v1 and v2, which brings
|
||||
2. Core integration was mentioned in scope for track 1 initially. This work has stalled a bit, largely due to organizational bandwidth and priorities. While there is a Nim Node API that in theory is ready to be used, having it be used in e.g. Status desktop or mobile app is a different matter. The team responsible for this at Status ([status-nim](github.com/status-im/status-nim)) has been making progress on getting nim-waku v1 integrated, and is expected to look into nim-waku v2 integration soon. One thing that makes this a especially tricky is the difference in interface between Waku v1 and v2, which brings
|
||||
us too...
|
||||
|
||||
3) Companion spec for encryption. As part of simplifying the protocol, the routing is decoupled from the encryption in v2 ([1](https://github.com/vacp2p/specs/issues/158), [2](https://github.com/vacp2p/specs/issues/181)). There are multiple layers of encryption at play here, and we need to figure out a design that makes sense for various use cases (dapps using Waku on their own, Status app, etc).
|
||||
3. Companion spec for encryption. As part of simplifying the protocol, the routing is decoupled from the encryption in v2 ([1](https://github.com/vacp2p/specs/issues/158), [2](https://github.com/vacp2p/specs/issues/181)). There are multiple layers of encryption at play here, and we need to figure out a design that makes sense for various use cases (dapps using Waku on their own, Status app, etc).
|
||||
|
||||
4) Bridge implementation. The spec is done and we know how it should work, but it needs to be implemented.
|
||||
4. Bridge implementation. The spec is done and we know how it should work, but it needs to be implemented.
|
||||
|
||||
5) General tightening up of specs and implementation.
|
||||
5. General tightening up of specs and implementation.
|
||||
|
||||
While this might seem like a lot, a lot has been done already, and the majority of the remaining tasks are more amendable to be pursued in parallel with other efforts. It is also worth mentioning that part of track 2 and 3 have been started, in the form of moving to GossipSub (amplification factors) and basics of adaptive nodes (multiple protocols). This is in addition to things like Waku Web which were not part of the initial scope.
|
||||
|
||||
|
|
@ -89,16 +89,16 @@ While this might seem like a lot, a lot has been done already, and the majority
|
|||
|
||||
Aside from the things mentioned above, what is coming up next? There are a few areas of interest, mentioned in no particular order. For track 2 and 3, see previous post for more details.
|
||||
|
||||
1) Better routing (track 2). While we are already building on top of GossipSub, we still need to explore things like topic sharding in more detail to further reduce amplification factors.
|
||||
1. Better routing (track 2). While we are already building on top of GossipSub, we still need to explore things like topic sharding in more detail to further reduce amplification factors.
|
||||
|
||||
2) Accounting and user-run nodes (track 3). With store and filter protocol getting ready, we can start to implement accounting and light connection game for incentivization in a bottom up and iterative manner.
|
||||
2. Accounting and user-run nodes (track 3). With store and filter protocol getting ready, we can start to implement accounting and light connection game for incentivization in a bottom up and iterative manner.
|
||||
|
||||
3) Privacy research. Study better and more rigorous privacy guarantees. E.g. how FloodSub/GossipSub behaves for common threat models, and how custom packet
|
||||
3. Privacy research. Study better and more rigorous privacy guarantees. E.g. how FloodSub/GossipSub behaves for common threat models, and how custom packet
|
||||
format can improve things like unlinkability.
|
||||
|
||||
4) zkSnarks RLN for spam protection and incentivization. We studied this [last year](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks) and recent developments have made this relevant to study again. Create an [experimental spec/PoC](https://github.com/vacp2p/specs/issues/189) as an extension to the relay protocol. Kudos to Barry Whitehat and others like Kobi Gurkan and Koh Wei Jie for pushing this!
|
||||
4. zkSnarks RLN for spam protection and incentivization. We studied this [last year](https://vac.dev/feasibility-semaphore-rate-limiting-zksnarks) and recent developments have made this relevant to study again. Create an [experimental spec/PoC](https://github.com/vacp2p/specs/issues/189) as an extension to the relay protocol. Kudos to Barry Whitehat and others like Kobi Gurkan and Koh Wei Jie for pushing this!
|
||||
|
||||
5) Ethereum M2M messaging. Being able to run in the browser opens up a lot of doors, and there is an opportunity here to enable things like a decentralized WalletConnect, multi-sig transactions, voting and similar use cases. This was the original goal of Whisper, and we'd like to deliver on that.
|
||||
5. Ethereum M2M messaging. Being able to run in the browser opens up a lot of doors, and there is an opportunity here to enable things like a decentralized WalletConnect, multi-sig transactions, voting and similar use cases. This was the original goal of Whisper, and we'd like to deliver on that.
|
||||
|
||||
As you can tell, quite a lot of thing! Luckily, we have two people joining as protocol engineers soon, which will bring much needed support for the current team of ~2-2.5 people. More details to come in further updates.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Problem No. 3 Inputs cryptocurrency key pair timestamp server, difficulty outputs! Key pair, halvening consensus "
|
||||
title: "Problem No. 3 Inputs cryptocurrency key pair timestamp server, difficulty outputs! Key pair, halvening consensus "
|
||||
name: 'Problem No. 3 Inputs cryptocurrency key pair timestamp server, difficulty outputs! Key pair, halvening consensus '
|
||||
title: 'Problem No. 3 Inputs cryptocurrency key pair timestamp server, difficulty outputs! Key pair, halvening consensus '
|
||||
date: 2020-11-10 12:00:00 +0800
|
||||
author: franck
|
||||
published: true
|
||||
permalink: /status-ipsum-3
|
||||
category: problem
|
||||
summary: "Status ipsum lorem exequoe"
|
||||
summary: 'Status ipsum lorem exequoe'
|
||||
image: /assets/img/js-waku-gist.png
|
||||
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/82
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,18 +1,18 @@
|
|||
---
|
||||
layout: post
|
||||
name: "[Talk] Vac, Waku v2 and Ethereum Messaging"
|
||||
title: "[Talk] Vac, Waku v2 and Ethereum Messaging"
|
||||
name: '[Talk] Vac, Waku v2 and Ethereum Messaging'
|
||||
title: '[Talk] Vac, Waku v2 and Ethereum Messaging'
|
||||
date: 2020-11-10 12:00:00 +0800
|
||||
author: oskarth
|
||||
published: true
|
||||
permalink: /waku-v2-ethereum-messaging
|
||||
categories: research
|
||||
category: research
|
||||
summary: Talk from Taipei Ethereum Meetup. Read on to find out about our journey from Whisper to Waku v2, as well as how Waku v2 can be useful for Etherum Messaging.
|
||||
image: /assets/img/taipei_ethereum_meetup_slide.png
|
||||
discuss: https://forum.vac.dev/t/discussion-talk-vac-waku-v2-and-ethereum-messaging/60
|
||||
---
|
||||
|
||||
*The following post is a transcript of the talk given at the [Taipei Ethereum meetup, November 5](https://www.meetup.com/Taipei-Ethereum-Meetup/events/274033344/). There is also a [video recording]( https://www.youtube.com/watch?v=lUDy1MoeYnI).*
|
||||
_The following post is a transcript of the talk given at the [Taipei Ethereum meetup, November 5](https://www.meetup.com/Taipei-Ethereum-Meetup/events/274033344/). There is also a [video recording](https://www.youtube.com/watch?v=lUDy1MoeYnI)._
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ date: 2021-03-05 12:00:00 +0800
|
|||
author: sanaztaheri
|
||||
published: true
|
||||
permalink: /rln-relay
|
||||
categories: research
|
||||
category: research
|
||||
summary: This post is going to give you an overview of how spam protection can be achieved in Waku Relay through rate-limiting nullifiers. We will cover a summary of spam-protection methods in centralized and p2p systems, and the solution overview and details of the economic spam-protection method. The open issues and future steps are discussed in the end.
|
||||
image: /assets/img/rain.png
|
||||
discuss: https://forum.vac.dev/t/privacy-preserving-p2p-economic-spam-protection-in-waku-v2-with-rate-limiting-nullfiers/66
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Problem No. 1 - Status ipsum lorem exequoe"
|
||||
title: "Problem No. 1 - Status ipsum lorem exequoe"
|
||||
name: 'Problem No. 1 - Status ipsum lorem exequoe'
|
||||
title: 'Problem No. 1 - Status ipsum lorem exequoe'
|
||||
date: 2021-03-05 12:00:00 +0800
|
||||
author: franck
|
||||
published: true
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
---
|
||||
layout: post
|
||||
name: "Presenting JS-Waku: Waku v2 in the Browser"
|
||||
title: "Presenting JS-Waku: Waku v2 in the Browser"
|
||||
name: 'Presenting JS-Waku: Waku v2 in the Browser'
|
||||
title: 'Presenting JS-Waku: Waku v2 in the Browser'
|
||||
date: 2021-06-04 12:00:00 +0800
|
||||
author: franck
|
||||
published: true
|
||||
permalink: /presenting-js-waku
|
||||
categories: platform
|
||||
summary: "JS-Waku is bringing Waku v2 to the browser. Learn what we achieved so far and what is next in our pipeline!"
|
||||
category: research
|
||||
summary: 'JS-Waku is bringing Waku v2 to the browser. Learn what we achieved so far and what is next in our pipeline!'
|
||||
image: /assets/img/js-waku-gist.png
|
||||
discuss: https://forum.vac.dev/t/discussion-presenting-js-waku-waku-v2-in-the-browser/81
|
||||
---
|
||||
|
|
@ -124,7 +124,10 @@ const waku = await Waku.create({});
|
|||
const nodes = await getStatusFleetNodes();
|
||||
await Promise.all(nodes.map((addr) => waku.dial(addr)));
|
||||
|
||||
const msg = WakuMessage.fromUtf8String("Here is a message!", "/my-cool-app/1/my-use-case/proto")
|
||||
const msg = WakuMessage.fromUtf8String(
|
||||
'Here is a message!',
|
||||
'/my-cool-app/1/my-use-case/proto'
|
||||
);
|
||||
await waku.relay.send(msg);
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue