status-im
/
specs
mirror of https://github.com/status-im/specs.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update/waku replace (#114) 

Introduces Waku in to the relevant places in the specs.

* First draft changes to 1-client.md

* initial waku replace work on 2-account.md

* Rename 3-whisper-usage to waku

There may be broken links after this change

* Initial update for 3-waku-usage.md

* Adjustment to use WHISPER concurrently with WAKU

* Converting 'moving away from whisper' section to `why use waku`

* Added clarification about Open Whisper Systems X3DH usage

* Updated transport privacy, include both whisper and waku

* Updated 1-client, language supports concurrent whisper and waku

* Updated 2-account.md to be whisper/waku concurrent

* Reverted 3-whisper-usage changes, added 9-waku-usage

* Complete revert of 3-whisper-usage

* revert 3-whisper-usage.md

* Fix to 3-whisper-usage.md links

* Update docs/stable/1-client.md

Co-authored-by: Dean Eigenmann <7621705+decanus@users.noreply.github.com>

* Removal of personal pronouns from 1-client.md

* Update to 5-secure-transport.md

* Implemented combined 'whispe/ waku relayer' role

* Addressed ambiguous and stale links in 6-payloads

* Updated 6-payloads

* Fix broken / stale links in 7-group-chat.md

* Fixed stale link 8-eips

* link fix and todo resolved for 1-client.md

* Added 10-waku-mailserver document

* Added gitignore

* Minor tweaks and fixes

* Added Whisper / Waku briding section to 3-whisper-usage

* Updated 10-waku-mailserver

* Updated 9-waku-usage.md

* Made changes to keep Mailserver term consistent

* Removed some redundant comments

* Removed seemingly vistigual '???'

* Including large payload PoW

See 7b6af0a7cc/protocol/message_processor.go (L25-L35)

* Changed 'mail server' to 'Mailserver'

* Added details about status-options

* Implemented final changes before review

* Tpyo fix

* Removed incorrect reference to topic-interest

* removed .gitignore, defer to global_gitignore

* Addressing feedback

* Revert draft/7 link fix

Readdressed in GH-120

* Reverted link fixes for stable/5

Addressing in GH-120

* Revert tpyo fix

Addressed in GH-119

* Reverted stable/6 link fixes

Addressed in GH-120

* Missed a minor link change

* Revert link fix in stable/1

Addressed in GH-120

* Revert tpyo fix in stable/1

Addressed in GH-119

* Revert stable/1 link fix

Addressed in GH-120

* Revert stable/2 link fix

Addressed in GH-120

* Revert link fix to stable/2

* Revert stable/3 link fixes

Addressed in GH-120

* Revert adding missing topic link

Addressed in GH-120

* Revert stable/4 link fix

Addressed in GH-120

* Revert stable/5 link fix

Addressed in GH-120

* Revert stable/8 link fix

Addressed in GH-120

* Partial revert of stable/1 node roles

* Resolved whispermail reference query

See GH-125

* Added changelog and version bump ...

Added my name where appropriate and addressed some minor issues

* Added consistent capitalisation to 'Whisper'

* Resolved jRPC call issue for waku

* Updated spelling

* Addressing minor feedback issues

* Update docs/stable/10-waku-mailserver.md

Co-authored-by: Dean Eigenmann <7621705+decanus@users.noreply.github.com>

* Update docs/stable/10-waku-mailserver.md

Co-authored-by: Dean Eigenmann <7621705+decanus@users.noreply.github.com>

Co-authored-by: Dean Eigenmann <7621705+decanus@users.noreply.github.com>
This commit is contained in:
Samuel Hawksby-Robinson 2020-05-22 15:05:21 +01:00 committed by GitHub
parent c22a9e8d5e
commit 664dd1c9df
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
9 changed files with 716 additions and 128 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

130
docs/stable/1-client.md
View File

@ -6,11 +6,11 @@ title: 1/CLIENT
# 1/CLIENT
> Version: 0.2
> Version: 0.3
>
> Status: Stable
>
> Authors: Adam Babik [adam@status.im](mailto:adam@status.im), Andrea Maria Piana [andreap@status.im](mailto:andreap@status.im), Dean Eigenmann [dean@status.im](mailto:dean@status.im), Corey Petty [corey@status.im](mailto:corey@status.im), Oskar Thorén [oskar@status.im](mailto:oskar@status.im) (alphabetical order)
> Authors: Adam Babik [adam@status.im](mailto:adam@status.im), Andrea Maria Piana [andreap@status.im](mailto:andreap@status.im), Dean Eigenmann [dean@status.im](mailto:dean@status.im), Corey Petty [corey@status.im](mailto:corey@status.im), Oskar Thorén [oskar@status.im](mailto:oskar@status.im), Samuel Hawksby-Robinson [samuel@status.im](mailto:samuel@status.im) (alphabetical order)
## Abstract
@ -36,7 +36,7 @@ have to be implemented in order to be a full Status client. The second gives a d
- [Bootstrapping](#bootstrapping)
- [Discovery](#discovery)
- [Mobile nodes](#mobile-nodes)
- [Transport privacy and Whisper usage](#transport-privacy-and-whisper-usage)
- [Transport privacy and Whisper/Waku usage](#transport-privacy-and-whisper--waku-usage)
- [Secure Transport](#secure-transport)
- [Data Sync](#data-sync)
- [Payloads and clients](#payloads-and-clients)
@ -47,10 +47,10 @@ have to be implemented in order to be a full Status client. The second gives a d
- [Why devp2p? Why not use libp2p?](#why-devp2p-why-not-use-libp2p)
- [What about other RLPx subprotocols like LES, and Swarm?](#what-about-other-rlpx-subprotocols-like-les-and-swarm)
- [Why do you use Whisper?](#why-do-you-use-whisper)
- [I heard you were moving away from Whisper?](#i-heard-you-were-moving-away-from-whisper)
- [Why is PoW for Whisper set so low?](#why-is-pow-for-whisper-set-so-low)
- [Why do you use Waku?](#why-do-you-use-waku)
- [Why is PoW for Waku set so low?](#why-is-pow-for-waku-set-so-low)
- [Why do you not use Discovery v5 for node discovery?](#why-do-you-not-use-discovery-v5-for-node-discovery)
- [I heard something about mailservers being trusted somehow?](#i-heard-something-about-mailservers-being-trusted-somehow)
- [I heard something about `Mailservers` being trusted somehow?](#i-heard-something-about-mailservers-being-trusted-somehow)
- [Data sync](#data-sync-1)
- [Why is MVDS not used for public chats?](#why-is-mvds-not-used-for-public-chats)
- [Footnotes](#footnotes)
@ -60,6 +60,8 @@ have to be implemented in order to be a full Status client. The second gives a d
- [Spam resistance](#spam-resistance)
- [Censorship resistance](#censorship-resistance)
- [Acknowledgments](#acknowledgments)
- [Changelog](#changelog)
- [Version 0.3](#version-03)
## Introduction
@ -74,7 +76,7 @@ Other aspects, such as how IPFS is used for stickers, how we interact with the E
| Data and payloads | End user functionality | 1:1, group chat, public chat |
| Data sync | Data consistency | MVDS Ratchet |
| Secure transport | Confidentiality, PFS, etc | Double Ratchet |
| Transport privacy | Routing, Metadata protection | Whisper |
| Transport privacy | Routing, Metadata protection | Waku / Whisper |
| P2P Overlay | Overlay routing, NAT traversal | devp2p |
### Protobuf
@ -95,22 +97,21 @@ To communicate between Status nodes, the [RLPx Transport
Protocol, v5](https://github.com/ethereum/devp2p/blob/master/rlpx.md) is used, which
allows for TCP-based communication between nodes.
On top of this we run the RLPx-based subprotocol [Whisper
v6](https://eips.ethereum.org/EIPS/eip-627) for privacy-preserving messaging.
There MUST be a node that is capable of discovering peers and
implements Whisper V6 specification.
On top of this we run RLPx-based subprotocols, the client
SHOULD NOT use [Whisper V6](https://eips.ethereum.org/EIPS/eip-627), the client
SHOULD use [Waku V1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md)
for privacy-preserving messaging and efficient usage of a node's bandwidth.
#### Node discovery and roles
There are four types of node roles:
1. Bootstrap nodes
2. Whisper relayers
3. Mailservers (servers and clients)
4. Mobile nodes (Status Clients)
1. `Bootstrap node`
1. `Whisper/Waku relayer`
1. `Mailserver` (servers and clients)
1. `Mobile node` (Status Clients)
To implement a standard Status client you MUST implement both 2. and 4. node types. The
other node types are optional, but we RECOMMEND you implement a mailserver
A standard Status client MUST implement both `Whisper/Waku relayer` and `Mobile node` node types. The
other node types are optional, but it is RECOMMEND to implement a `Mailserver`
client mode, otherwise the user experience is likely to be poor.
#### Bootstrapping
@ -119,7 +120,7 @@ To connect to other Status nodes you need to connect to a bootstrap node. These
nodes allow you to discover other nodes of the network.
Currently the main bootstrap nodes are provided by Status Gmbh, but anyone can
run these provided they are connected to the rest of the Whisper network.
run these provided they are connected to the rest of the Whisper/Waku network.
Status maintains a list of production fleet bootstrap nodes in the following locations:
@ -166,15 +167,15 @@ It uses Ethereum Node Records (ENR) to report discovered peers.
Both peers discovery mechanisms use topics to provide peers with certain capabilities.
There is no point in returning peers that do not support a particular protocol.
Status nodes that want to be discovered MUST register to Discovery V5 and/or Rendezvous
with the `whisper` topic. Status nodes that are mail servers and want to
with the `whisper` topic. Status nodes that are `Mailservers` and want to
be discoverable MUST additionally register with the `whispermail` topic.
It is RECOMMENDED to use both mechanisms but at the same time implement a structure
called `PeerPool`. `PeerPool` is responsible for maintaining an optimal number of peers.
For mobile nodes, there is no significant advantage to have more than 2-3 peers and one mail server.
For mobile nodes, there is no significant advantage to have more than 2-3 peers and one `Mailserver`.
`PeerPool` can notify peers discovery protocol implementations that they should suspend
their execution because the optimal number of peers is found. They should resume
if the number of connected peers drops or a mail server disconnects.
if the number of connected peers drops or a `Mailserver` disconnects.
It is worth noticing that an efficient caching strategy MAY be of great use, especially,
on mobile devices. Discovered peers can be cached as they rarely change and used
@ -194,30 +195,32 @@ the merkle tree found at [`fleets.status.im`](https://fleets.status.im), as desc
#### Mobile nodes
This is a Whisper node which connects to part of the Whisper network. It MAY
relay messages. See next section for more details on how to use Whisper to
communicate with other Status nodes.
A `Mobile node` is a Whisper and/or Waku node which connects to part of the respective Whisper
and/or Waku network(s). A `Mobile node` MAY relay messages. See next section for more details on how
to use Whisper and/or Waku to communicate with other Status nodes.
### Transport privacy and Whisper usage
### Transport privacy and Whisper / Waku usage
Once a Whisper node is up and running there are some specific settings required
Once a Whisper and/or Waku node is up and running there are some specific settings required
to communicate with other Status nodes.
See [3/WHISPER-USAGE](https://specs.status.im/spec/3) for more details.
See [3/WHISPER-USAGE](https://specs.status.im/spec/3) and [9/WAKU-USAGE](https://specs.status.im/spec/9) for more details.
For providing an offline inbox, see the complementary [4/WHISPER-MAILSERVER](https://specs.status.im/spec/4).
For providing an offline inbox, see the complementary [4/WHISPER-MAILSERVER](https://specs.status.im/spec/4) and [10/WAKU-MAILSERVER](https://specs.status.im/spec/10).
### Secure Transport
In order to provide confidentiality, integrity, authentication and forward
secrecy of messages we implement a secure transport on top of Whisper. This is
secrecy of messages we implement a secure transport on top of Whisper and Waku. This is
used in 1:1 chats and group chats, but not for public chats. See [5/SECURE-TRANSPORT](https://specs.status.im/spec/5) for more.
### Data Sync
[MVDS](https://specs.vac.dev/mvds.html) is used for 1:1 and group chats, however it is currently not in use for public chats.
[Status payloads](#payloads-and-clients) are serialized and then wrapped inside a MVDS message which is added to an [MVDS payload](https://specs.vac.dev/mvds.html#payloads), this payload is then encrypted (if necessary for 1-to-1 / group-chats) and sent using whisper which also encrypts it.
[Status payloads](#payloads-and-clients) are serialized and then wrapped inside an
MVDS message which is added to an [MVDS payload](https://specs.vac.dev/mvds.html#payloads),
this payload is then encrypted (if necessary for 1-to-1 / group-chats) and sent using
Whisper or Waku which also encrypts it.
### Payloads and clients
@ -270,20 +273,30 @@ vision of Ethereum as the world computer, Ethereum and Swarm being the other
two. Status was started as an encapsulation of and a clear window to this world
computer.
#### I heard you were moving away from Whisper?
#### Why do you use Waku?
Whisper is not currently under active development, and it has several drawbacks.
Among others:
Waku is a direct upgrade and replacement for Whisper, the main motivation for
developing and implementing Waku can be found in the [Waku specs](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#motivation).
- It is very wasteful bandwidth-wise and it doesn't appear to be scalable
- Proof of work is a poor spam protection mechanism for heterogeneous devices
- The privacy guarantees provided are not rigorous
- There's no incentives to run a node
>Waku was created to incrementally improve in areas that Whisper is lacking in,
>with special attention to resource restricted devices. We specify the standard for
>Waku messages in order to ensure forward compatibility of different Waku clients,
>backwards compatibility with Whisper clients, as well as to allow multiple
>implementations of Waku and its capabilities. We also modify the language to be more
>unambiguous, concise and consistent.
Considerable work has gone into the active development of Ethereum, in contrast Whisper
is not currently under active development, and it has several drawbacks. Among others:
- Whisper is very wasteful bandwidth-wise and doesn't appear to be scalable
- Proof of work is a poor spam protection mechanism for heterogeneous devices
- The privacy guarantees provided are not rigorous
- There are no incentives to run a node
Finding a more suitable transport privacy is an ongoing research effort,
together with [Vac](https://vac.dev/vac-overview) and other teams in the space.
#### Why is PoW for Whisper set so low?
#### Why is PoW for Waku set so low?
A higher PoW would be desirable, but this kills the battery on mobile phones,
which is a prime target for Status clients.
@ -301,9 +314,9 @@ Discovery v5 for a short period of time until their peer list is populated.
For some further investigation, see
[here](https://github.com/status-im/swarms/blob/master/ideas/092-disc-v5-research.md).
#### I heard something about mailservers being trusted somehow?
#### I heard something about `Mailservers` being trusted somehow?
In order to use a mail server, a given node needs to connect to it directly, i.e. add the mailserver as its peer and mark it as trusted. This means that the mail server is able to send direct p2p messages to the node instead of broadcasting them. Effectively, it knows the bloom filter of the topics the node is interested in, when it is online as well as many metadata like IP address.
In order to use a `Mailserver`, a given node needs to connect to it directly, i.e. add the `Mailserver` as its peer and mark it as trusted. This means that the `Mailserver` is able to send direct p2p messages to the node instead of broadcasting them. Effectively, it knows the bloom filter of the topics the node is interested in, when it is online as well as many metadata like IP address.
### Data sync
@ -315,8 +328,7 @@ whereby participants can sync. Additionally, MVDS is currently not optimized for
large group contexts, which means bandwidth usage will be a lot higher than
reasonable. See [P2P Data Sync for
Mobile](https://vac.dev/p2p-data-sync-for-mobile) for more. This is an active
area of research. The bandwidth issue is further exacerbated by Whisper being
very bandwidth heavy.
area of research.
## Footnotes
@ -326,17 +338,17 @@ very bandwidth heavy.
## Appendix A: Security considerations
There are several security considerations to take into account when running Status. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used, such as mailserver, light node, and so on.
There are several security considerations to take into account when running Status. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used, such as `Mailserver`, light node, and so on.
### Scalability and UX
**Bandwidth usage:**
In version 1 of Status, bandwidth usage is likely to be an issue. For more investigation into this, see [the theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
In version 1 of Status, bandwidth usage is likely to be an issue. In Status version 1.1 this is partially addressed with Waku usage, see [the theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
**Mailserver High Availability requirement:**
**`Mailserver` High Availability requirement:**
A mailserver has to be online to receive messages for other nodes, this puts a high availability requirement on it.
A `Mailserver` has to be online to receive messages for other nodes, this puts a high availability requirement on it.
**Gossip-based routing:**
@ -356,13 +368,13 @@ The main privacy concern with light nodes is that directly connected peers will
By having a bloom filter where only the topics you are interested in are set, you reveal which messages you are interested in. This is a fundamental trade-off between bandwidth usage and privacy, though the trade-off space is likely suboptimal in terms of the [Anonymity](https://eprint.iacr.org/2017/954.pdf) [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
**Mailserver client privacy:**
**`Mailserver client` privacy:**
A mailserver client has to trust a mailserver, which means they can send direct traffic. This reveals what topics / bloom filter a node is interested in, along with its peerID (with IP).
A `Mailserver client` has to trust a `Mailserver`, which means they can send direct traffic. This reveals what topics / bloom filter a node is interested in, along with its peerID (with IP).
**Privacy guarantees not rigorous:**
Privacy for Whisper hasn't been studied rigorously for various threat models like global passive adversary, local active attacker, etc. This is unlike e.g. Tor and mixnets.
Privacy for Whisper or Waku hasn't been studied rigorously for various threat models like global passive adversary, local active attacker, etc. This is unlike e.g. Tor and mixnets.
**Topic hygiene:**
@ -374,9 +386,9 @@ Similar to bloom filter privacy, if you use a very specific topic you reveal mor
Proof of work is a poor spam prevention mechanism. A mobile device can only have a very low PoW in order not to use too much CPU / burn up its phone battery. This means someone can spin up a powerful node and overwhelm the network.
**Mailserver trusted connection:**
**`Mailserver` trusted connection:**
A mailserver has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning mailserver can overwhelm an individual node.
A `Mailserver` has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning `Mailserver` can overwhelm an individual node.
### Censorship resistance
@ -389,3 +401,15 @@ See <https://github.com/status-im/status-react/issues/6351> for some discussion.
## Acknowledgments
Jacek Sieka
## Changelog
### Version 0.3
Released `TODO`
- Added that Waku SHOULD be used
- Added that Whisper SHOULD NOT be used
- Added language to include Waku in all relevant places
- Change to keep `Mailserver` term consistent

129
docs/stable/10-waku-mailserver.md Normal file
View File

@ -0,0 +1,129 @@
---
permalink: /spec/10
parent: Stable specs
title: 10/WAKU-MAILSERVER
---
# 10/WAKU-MAILSERVER
> Version: 0.1
>
> Status: Stable
>
> Authors: Adam Babik <adam@status.im>, Oskar Thorén <oskar@status.im>, Samuel Hawksby-Robinson <samuel@status.im> (alphabetical order)
- [Status Waku Mailserver Specification](#10waku-mailserver)
- [Abstract](#abstract)
- [`Mailserver`](#mailserver)
- [Archiving messages](#archiving-messages)
- [Requesting messages](#requesting-messages)
- [Receiving historic messages](#receiving-historic-messages)
- [Security considerations](#security-considerations)
- [Confidentiality](#confidentiality)
- [Altruistic and centralized operator risk](#altruistic-and-centralized-operator-risk)
- [Privacy concerns](#privacy-concerns)
- [Denial-of-service](#denial-of-service)
- [Changelog](#changelog)
- [Version 0.1](#version-01)
## Abstract
Being mostly offline is an intrinsic property of mobile clients. They need to save network transfer and battery consumption to avoid spending too much money or constant charging. Waku protocol, on the other hand, is an online protocol. Messages are available in the Waku network only for short period of time calculate in seconds.
Waku Mailserver is a specification that allows messages to be stored permanently and to allows the stored messages to be delivered to requesting client nodes, regardless if the messages are not available in the network due to the message TTL expiring.
## `Mailserver`
From the network perspective, a `Mailserver` is just like any other Waku node. The only difference is that a `Mailserver` has the capability of archiving messages and delivering them to its peers on-demand.
It is important to notice that a `Mailserver` will only handle requests from its direct peers and exchanged packets between a `Mailserver` and a peer are p2p messages.
### Archiving messages
A node which wants to provide `Mailserver` functionality MUST store envelopes from
incoming message packets (Waku packet-code `0x01`). The envelopes can be stored in any
format, however they MUST be serialized and deserialized to the Waku envelope format.
A `Mailserver` SHOULD store envelopes for all topics to be generally useful for any peer, however for specific use cases it MAY store envelopes for a subset of topics.
### Requesting messages
In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing `Mailserver` functionality. This packet requires one argument which MUST be a Waku envelope.
In the Waku envelope's payload section, there MUST be RLP-encoded information about the details of the request:
```
[ Lower, Upper, Bloom, Limit, Cursor ]
```
`Lower`: 4-byte wide unsigned integer (UNIX time in seconds; oldest requested envelope's creation time)
`Upper`: 4-byte wide unsigned integer (UNIX time in seconds; newest requested envelope's creation time)
`Bloom`: 64-byte wide array of Waku topics encoded in a bloom filter to filter envelopes
`Limit`: 4-byte wide unsigned integer limiting the number of returned envelopes
`Cursor`: an array of a cursor returned from the previous request (optional)
The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit` so that the requester can send another request using the obtained `Cursor` value. What exactly is in the `Cursor` is up to the implementation. The requester SHOULD NOT use a `Cursor` obtained from one `Mailserver` in a request to another `Mailserver` because the format or the result MAY be different.
The envelope MUST be encrypted with a symmetric key agreed between the requester and the `Mailserver`.
### Receiving historic messages
Historic messages MUST be sent to a peer as a packet with a P2P Message code (`0x7f`) followed by an array of Waku envelopes.
In order to receive historic messages from a `Mailserver`, a node MUST trust the selected `Mailserver`, that is allow to receive packets with the P2P Message code. By default, such packets are discarded.
Received envelopes MUST be passed through the Waku envelope pipelines so that they are picked up by registered filters and passed to subscribers.
For a requester, to know that all messages have been sent by a `Mailserver`, it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by the following parameters:
```
[ RequestID, LastEnvelopeHash, Cursor ]
```
- `RequestID`: 32-byte wide array with a Keccak-256 hash of the envelope containing the original request
- `LastEnvelopeHash`: 32-byte wide array with a Keccak-256 hash of the last sent envelope for the request
- `Cursor`: an array of a cursor returned from the previous request (optional)
If `Cursor` is not empty, it means that not all messages were sent due to the set `Limit` in the request. One or more consecutive requests MAY be sent with `Cursor` field filled in order to receive the rest of the messages.
## Security considerations
### Confidentiality
All Waku envelopes are encrypted. A `Mailserver` node can not inspect their contents.
### Altruistic and centralized operator risk
In order to be useful, a `Mailserver` SHOULD be online most of time. That means
users either have to be a bit tech-savvy to run their own node, or rely on someone
else to run it for them.
Currently one of Status's legal entities provides `Mailservers` in an altruistic manner, but this is
suboptimal from a decentralization, continuance and risk point of view. Coming
up with a better system for this is ongoing research.
A Status client SHOULD allow the `Mailserver` selection to be customizable.
### Privacy concerns
In order to use a `Mailserver`, a given node needs to connect to it directly,
i.e. add the `Mailserver` as its peer and mark it as trusted. This means that the
`Mailserver` is able to send direct p2p messages to the node instead of
broadcasting them. Effectively, it will have access to the bloom filter of
topics that the user is interested in, when it is online as well as many
metadata like IP address.
### Denial-of-service
Since a `Mailserver` is delivering expired envelopes and has a direct TCP connection with the recipient, the recipient is vulnerable to DoS attacks from a malicious `Mailserver` node.
## Changelog
### Version 0.1
Released `TODO`
- Created document
- Forked from [4-whisper-mailserver](4-whisper-mailserver.md)
- Change to keep `Mailserver` term consistent
- Replaced Whisper references with Waku

54
docs/stable/2-account.md
View File

@ -6,11 +6,11 @@ title: 2/ACCOUNT
# 2/ACCOUNT
> Version: 0.2
> Version: 0.3
>
> Status: Stable
>
> Authors: Corey Petty <corey@status.im>, Oskar Thorén <oskar@status.im> (alphabetical order)
> Authors: Corey Petty <corey@status.im>, Oskar Thorén <oskar@status.im>, Samuel Hawksby-Robinson <samuel@status.im> (alphabetical order)
## Abstract
@ -38,16 +38,18 @@ In this specification we explain what Status account is, and how trust is establ
- [Bundles](#bundles)
- [Contact Verification](#contact-verification)
- [Identicon](#identicon)
- [3 word pseudonym / whisper key fingerprint](#3-word-pseudonym--whisper-key-fingerprint)
- [3 word pseudonym / Whisper/Waku key fingerprint](#3-word-pseudonym--whisperwaku-key-fingerprint)
- [ENS name](#ens-name)
- [Security Considerations](#security-considerations)
- [Changelog](#changelog)
- [Version 0.3](#version-03)
<!-- markdown-toc end -->
## Introduction
The core concept of an account in Status is a set of cryptographic keypairs. Namely, the combination of the following:
1. a whisper chat identity keypair
1. a Whisper/Waku chat identity keypair
1. a set of cryptocurrency wallet keypairs
Everything else associated with the contact is either verified or derived from the above items, including:
@ -60,18 +62,18 @@ Everything else associated with the contact is either verified or derived from t
### Public/Private Keypairs
- An ECDSA (secp256k1 curve) public/private keypair MUST be generated via a [BIP43](https://github.com/bitcoin/bips/blob/master/bip-0043.mediawiki) derived path from a [BIP39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic seed phrase.
- The default paths are defined as such:
- Whisper Chat Key (`IK`): `m/43'/60'/1581'/0'/0` (post Multiaccount integration)
- Whisper/Waku Chat Key (`IK`): `m/43'/60'/1581'/0'/0` (post Multiaccount integration)
- following [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md)
<!-- WE CURRENTLY DO NOT IMPLEMENT ENCRYPTION KEY, FOR FUTURE - C.P. -->
<!-- - DB encryption Key (`DBK`): `m/43'/60'/1581'/1'/0` (post Multiaccount integration)
- following [EIP1581](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md) -->
- Status Wallet paths: `m/44'/60'/0'/0/i` starting at `i=0`
- following [BIP44](https://github.com/bitcoin/bips/blob/master/bip-0044.mediawiki)
- NOTE: this (`i=0`) is also the current (and only) path for Whisper key before Multiaccount integration
- NOTE: this (`i=0`) is also the current (and only) path for Whisper/Waku key before Multiaccount integration
### X3DH Prekey bundle creation
- Status follows the X3DH prekey bundle scheme that Open Whisper Systems outlines [in their documentation](https://signal.org/docs/specifications/x3dh/#the-x3dh-protocol) with the following exceptions:
- Because there are no central servers, we do not publish one-time keys `OPK` or perform DH including them.
- Status follows the X3DH prekey bundle scheme that [Open Whisper Systems](https://en.wikipedia.org/wiki/Signal_Messenger#2013%E2%80%932018:_Open_Whisper_Systems) (not to be confused with the Whisper sub-protocol) outlines [in their documentation](https://signal.org/docs/specifications/x3dh/#the-x3dh-protocol) with the following exceptions:
- Status does not publish one-time keys `OPK` or perform DH including them, because there are no central servers in the Status implementation.
- A client MUST create X3DH prekey bundles, each defined by the following items:
- Identity Key: `IK`
- Signed prekey: `SPK`
@ -84,14 +86,14 @@ Everything else associated with the contact is either verified or derived from t
### X3DH Prekey bundles
- A client SHOULD regenerate a new X3DH prekey bundle every 24 hours. This MAY be done in a lazy way, such that a client that does not come online past this time period does not regenerate or broadcast bundles.
- The current bundle SHOULD be broadcast on a whisper topic specific to his Identity Key, `{IK}-contact-code`, intermittently. This MAY be done every 6 hours.
- The current bundle SHOULD be broadcast on a Whisper/Waku topic specific to his Identity Key, `{IK}-contact-code`, intermittently. This MAY be done every 6 hours.
- A bundle SHOULD accompany every message sent.
- TODO: retrieval of long-time offline users bundle via `{IK}-contact-code`
## Optional Account additions
### ENS Username
- A user MAY register a public username on the Ethereum Name System (ENS). This username is a user-chosen subdomain of the `stateofus.eth` ENS registration that maps to their whisper identity key (`IK`).
- A user MAY register a public username on the Ethereum Name System (ENS). This username is a user-chosen subdomain of the `stateofus.eth` ENS registration that maps to their Whisper/Waku identity key (`IK`).
<!-- ### User Profile Picture
- An account MAY edit the `IK` generated identicon with a chosen picture. This picture will become part of the publicly broadcast profile of the account. -->
@ -106,10 +108,10 @@ Everything else associated with the contact is either verified or derived from t
### Terms Glossary
| term | description |
| ---- | ----------- |
| ---------------- | ----------- |
| privkey | ECDSA secp256k1 private key |
| pubkey | ECDSA secp256k1 public key |
| whisper key | pubkey for chat with HD derivation path m/43'/60'/1581'/0'/0 |
| Whisper/Waku key | pubkey for chat with HD derivation path m/43'/60'/1581'/0'/0 |
### Contact Discovery
@ -133,8 +135,8 @@ This can be done in the following ways:
1. public key via public channel listening
- `status-react/src/status_im/contact_code/core.cljs`
1. contact codes
2. decentralized storage (not implemented)
3. whisper
1. decentralized storage (not implemented)
1. Whisper/Waku
### Initial Key Exchange
@ -148,24 +150,24 @@ This can be done in the following ways:
Signature // Prekey signature
Timestamp // When the bundle was lasted created locally
```
- include BundleContainer???
- include BundleContainer
- a new bundle SHOULD be created at least every 12 hours
- a bundle is only generated when it is used
- a bundle SHOULD be distributed on the contact code channel. This is the whisper topic `{IK}-contact-code`, where `IK` is the hex encoded public key of the user, prefixed with `0x`. The channel is encrypted in the same way public chats are encrypted.
- a bundle SHOULD be distributed on the contact code channel. This is the Whisper and Waku topic `{IK}-contact-code`, where `IK` is the hex encoded public key of the user, prefixed with `0x`. The channel is encrypted in the same way public chats are encrypted.
### Contact Verification
Once you have the information of a contact, the following can be used to verify that the key material is as it should be.
#### Identicon
A low-poly identicon is deterministically generated from the whisper chat public key. This can then be compared out of band to ensure the receiver's public key is the one you have locally.
A low-poly identicon is deterministically generated from the Whisper/Waku chat public key. This can then be compared out of band to ensure the receiver's public key is the one you have locally.
#### 3 word pseudonym / whisper key fingerprint
Status generates a deterministic 3-word random pseudonym from the whisper chat public key. This pseudonym acts as a human readable fingerprint to the whisper chat public key. This name also shows when viewing a contact's public profile and in the chat UI.
#### 3 word pseudonym / Whisper/Waku key fingerprint
Status generates a deterministic 3-word random pseudonym from the Whisper/Waku chat public key. This pseudonym acts as a human readable fingerprint to the Whisper/Waku chat public key. This name also shows when viewing a contact's public profile and in the chat UI.
- implementation: [gfycat](https://github.com/status-im/status-react/tree/develop/src/status_im/utils/gfycat)
#### ENS name
Status offers the ability to register a mapping of a human readable subdomain of `stateofus.eth` to their whisper chat public key. This registration is purchased (currently by staking 10 SNT) and stored on the Ethereum mainnet blockchain for public lookup.
Status offers the ability to register a mapping of a human readable subdomain of `stateofus.eth` to their Whisper/Waku chat public key. This registration is purchased (currently by staking 10 SNT) and stored on the Ethereum mainnet blockchain for public lookup.
<!-- TODO: Elaborate on security implications -->
@ -183,7 +185,7 @@ possible connections
- public chat
- client - mailserver (statusd + ???)
- a mailserver identifies itself by an [enode address](https://github.com/ethereum/wiki/wiki/enode-url-format)
- client - whisper node (statusd)
- client - Whisper/Waku node (statusd)
- a node identifies itself by an enode address
- client - bootnode (go-ethereum)
- a bootnode identifies itself by
@ -217,3 +219,13 @@ All messages sent are encrypted with the public key of the destination and signe
## Security Considerations
-
## Changelog
### Version 0.3
Released `TODO`
- Added language to include Waku in all relevant places
- Change to keep `Mailserver` term consistent
- Added clarification to Open Whisper Systems

41
docs/stable/3-whisper-usage.md
View File

@ -6,11 +6,11 @@ title: 3/WHISPER-USAGE
# 3/WHISPER-USAGE
> Version: 0.2
> Version: 0.3
>
> Status: Stable
>
> Authors: Adam Babik <adam@status.im>, Corey Petty <corey@status.im>, Oskar Thorén <oskar@status.im> (alphabetical order)
> Authors: Adam Babik <adam@status.im>, Corey Petty <corey@status.im>, Oskar Thorén <oskar@status.im>, Samuel Hawksby-Robinson <samuel@status.im> (alphabetical order)
- [Abstract](#abstract)
- [Reason](#reason)
@ -26,9 +26,12 @@ title: 3/WHISPER-USAGE
- [Group chat topic](#group-chat-topic)
- [Message encryption](#message-encryption)
- [Message confirmations](#message-confirmations)
- [Whisper / Waku bridging](#whisper--waku-bridging)
- [Whisper V6 extensions](#whisper-v6-extensions)
- [Request historic messages](#request-historic-messages)
- [shhext_requestMessages](#shhext_requestmessages)
- [Changelog](#changelog)
- [Version 0.3](#version-03)
## Abstract
@ -308,15 +311,28 @@ The supported codes:
The drawback of sending message confirmations is that it increases the noise in the network because for each sent message, a corresponding confirmation is broadcast by one or more peers. To limit that, both Batch Acknowledge packet (`0x0b`) and Message Response packet (`0x0c`) are not broadcast to peers of the peers, i.e. they do not follow epidemic spread.
In the current Status network setup, only Mailservers support message confirmations. A client posting a message to the network and after receiving a confirmation can be sure that the message got processed by the Mailserver. If additionally, sending a message is limited to non-Mailserver peers, it also guarantees that the message got broadcast through the network and it reached the selected Mailserver.
In the current Status network setup, only `Mailservers` support message confirmations. A client posting a message to the network and after receiving a confirmation can be sure that the message got processed by the `Mailserver`. If additionally, sending a message is limited to non-`Mailserver` peers, it also guarantees that the message got broadcast through the network and it reached the selected `Mailserver`.
## Whisper / Waku bridging
In order to maintain compatibility between Whisper and Waku nodes, a Status network that
implements both Whisper and Waku messaging protocols MUST have at least one node that is
capable of discovering peers and implements
[Whisper v6](https://eips.ethereum.org/EIPS/eip-627),
[Waku V0](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-0.md) and
[Waku V1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md) specifications.
Additionally, any Status network that implements both Whisper and Waku messaging protocols
MUST implement bridging capabilities as detailed in
[Waku V1#Bridging](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#waku-whisper-bridging).
## Whisper V6 extensions
### Request historic messages
Sends a request for historic messages to a Mailserver. The Mailserver node MUST be a direct peer and MUST be marked as trusted (using `shh_markTrustedPeer`).
Sends a request for historic messages to a `Mailserver`. The `Mailserver` node MUST be a direct peer and MUST be marked as trusted (using `shh_markTrustedPeer`).
The request does not wait for the response. It merely sends a peer-to-peer message to the Mailserver and it's up to Mailserver to process it and start sending historic messages.
The request does not wait for the response. It merely sends a peer-to-peer message to the `Mailserver` and it's up to `Mailserver` to process it and start sending historic messages.
The drawback of this approach is that it is impossible to tell which historic messages are the result of which request.
@ -326,17 +342,26 @@ It's recommended to return messages from newest to oldest. To move further back
**Parameters**:
1. Object - The message request object:
* `mailServerPeer` - `String`: Mailserver's enode address.
* `mailServerPeer` - `String`: `Mailserver`'s enode address.
* `from` - `Number` (optional): Lower bound of time range as unix timestamp, default is 24 hours back from now.
* `to` - `Number` (optional): Upper bound of time range as unix timestamp, default is now.
* `limit` - `Number` (optional): Limit the number of messages sent back, default is no limit.
* `cursor` - `String` (optional): Used for paginated requests.
* `topics` - `Array`: hex-encoded message topics.
* `symKeyID` - `String`: an ID of a symmetric key to authenticate to Mailserver, derived from Mailserver password.
* `symKeyID` - `String`: an ID of a symmetric key used to authenticate with the `Mailserver`, derived from Mailserver password.
**Returns**:
`Boolean` - returns `true` if the request was sent.
The above `topics` is then converted into a bloom filter and then and sent to the Mailserver.
The above `topics` is then converted into a bloom filter and then and sent to the `Mailserver`.
<!-- TODO: Clarify actual request with bloom filter to mailserver -->
## Changelog
### Version 0.3
Released `TODO`
- Added Whisper / Waku Bridging section
- Change to keep `Mailserver` term consistent

52
docs/stable/4-whisper-mailserver.md
View File

@ -6,14 +6,14 @@ title: 4/WHISPER-MAILSERVER
# 4/WHISPER-MAILSERVER
> Version: 0.2
> Version: 0.3
>
> Status: Stable
>
> Authors: Adam Babik <adam@status.im>, Oskar Thorén <oskar@status.im> (alphabetical order)
- [Abstract](#abstract)
- [Mailserver](#mailserver)
- [`Mailserver`](#mailserver)
- [Archiving messages](#archiving-messages)
- [Requesting messages](#requesting-messages)
- [Receiving historic messages](#receiving-historic-messages)
@ -22,28 +22,30 @@ title: 4/WHISPER-MAILSERVER
- [Altruistic and centralized operator risk](#altruistic-and-centralized-operator-risk)
- [Privacy concerns](#privacy-concerns)
- [Denial-of-service](#denial-of-service)
- [Changelog](#changelog)
- [Version 0.3](#version-03)
## Abstract
Being mostly offline is an intrinsic property of mobile clients. They need to save network transfer and battery consumption to avoid spending too much money or constant charging. Whisper protocol, on the other hand, is an online protocol. Messages are available in the Whisper network only for short period of time calculate in seconds.
Whisper Mailserver is a Whisper extension that allows to store messages permanently and deliver them to the clients even though they are already not available in the network and expired.
Whisper `Mailserver` is a Whisper extension that allows to store messages permanently and deliver them to the clients even though they are already not available in the network and expired.
## Mailserver
## `Mailserver`
From the network perspective, Mailserver is just like any other Whisper node. The only difference is that it has a capability of archiving messages and delivering them to its peers on-demand.
From the network perspective, `Mailserver` is just like any other Whisper node. The only difference is that it has a capability of archiving messages and delivering them to its peers on-demand.
It is important to notice that Mailserver will only handle requests from its direct peers and exchanged packets between Mailserver and a peer are p2p messages.
It is important to notice that `Mailserver` will only handle requests from its direct peers and exchanged packets between `Mailserver` and a peer are p2p messages.
### Archiving messages
A node which wants to provide mailserver functionality MUST store envelopes from incoming message packets (Whisper packet-code `0x01`). The envelopes can be stored in any format, however they MUST be serialized and deserialized to the Whisper envelope format.
A node which wants to provide `Mailserver` functionality MUST store envelopes from incoming message packets (Whisper packet-code `0x01`). The envelopes can be stored in any format, however they MUST be serialized and deserialized to the Whisper envelope format.
A mailserver SHOULD store envelopes for all topics to be generally useful for any peer, however for specific use cases it MAY store envelopes for a subset of topics.
A `Mailserver` SHOULD store envelopes for all topics to be generally useful for any peer, however for specific use cases it MAY store envelopes for a subset of topics.
### Requesting messages
In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing mailserver functionality. This packet requires one argument which MUST be a Whisper envelope.
In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing `Mailserver` functionality. This packet requires one argument which MUST be a Whisper envelope.
In the Whisper envelope's payload section, there MUST be RLP-encoded information about the details of the request:
@ -57,19 +59,19 @@ In the Whisper envelope's payload section, there MUST be RLP-encoded information
`Limit`: 4-byte wide unsigned integer limiting the number of returned envelopes
`Cursor`: an array of a cursor returned from the previous request (optional)
The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit` so that the requester can send another request using the obtained `Cursor` value. What exactly is in the `Cursor` is up to the implementation. The requester SHOULD NOT use a `Cursor` obtained from one mailserver in a request to another mailserver because the format or the result MAY be different.
The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit` so that the requester can send another request using the obtained `Cursor` value. What exactly is in the `Cursor` is up to the implementation. The requester SHOULD NOT use a `Cursor` obtained from one `Mailserver` in a request to another `Mailserver` because the format or the result MAY be different.
The envelope MUST be encrypted with a symmetric key agreed between the requester and Mailserver.
The envelope MUST be encrypted with a symmetric key agreed between the requester and `Mailserver`.
### Receiving historic messages
Historic messages MUST be sent to a peer as a packet with a P2P Message code (`0x7f`) followed by an array of Whisper envelopes. It is incompatible with the original Whisper spec (EIP-627) because it allows only a single envelope, however, an array of envelopes is much more performant. In order to stay compatible with EIP-627, a peer receiving historic message MUST handle both cases.
In order to receive historic messages from a mailserver, a node MUST trust the selected mailserver, that is allow to receive packets with the P2P Message code. By default, such packets are discarded.
In order to receive historic messages from a `Mailserver`, a node MUST trust the selected `Mailserver`, that is allow to receive packets with the P2P Message code. By default, such packets are discarded.
Received envelopes MUST be passed through the Whisper envelope pipelines so that they are picked up by registered filters and passed to subscribers.
For a requester, to know that all messages have been sent by mailserver, it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by the following parameters:
For a requester, to know that all messages have been sent by `Mailserver`, it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by the following parameters:
```
[ RequestID, LastEnvelopeHash, Cursor ]
@ -85,29 +87,37 @@ If `Cursor` is not empty, it means that not all messages were sent due to the se
### Confidentiality
All Whisper envelopes are encrypted. Mailserver node can not inspect their contents.
All Whisper envelopes are encrypted. A `Mailserver` node can not inspect their contents.
### Altruistic and centralized operator risk
In order to be useful, a mailserver SHOULD be online most of time. That means
In order to be useful, a `Mailserver` SHOULD be online most of time. That means
you either have to be a bit tech-savvy to run your own node, or rely on someone
else to run it for you.
Currently one of Status's legal entities provides mailservers in an altruistic manner, but this is
Currently one of Status's legal entities provides `Mailservers` in an altruistic manner, but this is
suboptimal from a decentralization, continuance and risk point of view. Coming
up with a better system for this is ongoing research.
A Status client SHOULD allow the mailserver selection to be customizable.
A Status client SHOULD allow the `Mailserver` selection to be customizable.
### Privacy concerns
In order to use a Mailserver, a given node needs to connect to it directly,
i.e. add the Mailserver as its peer and mark it as trusted. This means that the
Mailserver is able to send direct p2p messages to the node instead of
In order to use a `Mailserver`, a given node needs to connect to it directly,
i.e. add the `Mailserver` as its peer and mark it as trusted. This means that the
`Mailserver` is able to send direct p2p messages to the node instead of
broadcasting them. Effectively, it will have access to the bloom filter of
topics that the user is interested in, when it is online as well as many
metadata like IP address.
### Denial-of-service
Since a mailserver is delivering expired envelopes and has a direct TCP connection with the recipient, the recipient is vulnerable to DoS attacks from a malicious mailserver node.
Since a `Mailserver` is delivering expired envelopes and has a direct TCP connection with the recipient, the recipient is vulnerable to DoS attacks from a malicious `Mailserver` node.
## Changelog
### Version 0.3
Released `TODO`
- Change to keep `Mailserver` term consistent

38
docs/stable/5-secure-transport.md
View File

@ -6,7 +6,7 @@ title: 5/SECURE-TRANSPORT
# 5/SECURE-TRANSPORT
> Version: 0.2
> Version: 0.3
>
> Status: Stable
>
@ -49,6 +49,8 @@ It builds on the [X3DH](https://signal.org/docs/specifications/x3dh/) and [Doubl
- [Sending messages to a paired group](#sending-messages-to-a-paired-group)
- [Account recovery](#account-recovery)
- [Partitioned devices](#partitioned-devices)
- [Changelog](#changelog)
- [Version 0.3](#version-03)
## Introduction
@ -75,7 +77,7 @@ Types used in this specification are defined using [Protobuf](https://developers
### Transport Layer
[Whisper](./3-whisper-usage.md) serves as the transport layer for the Status chat protocol.
[Whisper](3-whisper-usage.md) and [Waku](9-waku-usage.md) serves as the transport layers for the Status chat protocol.
### User flow for 1-to-1 communications
@ -87,7 +89,7 @@ See [Account specification](./2-account.md)
If Alice later recovers her account, the Double Ratchet state information will not be available, so she is no longer able to decrypt any messages received from existing contacts.
If an incoming message (on the same Whisper topic) fails to decrypt, a message is replied with the current bundle, so that the other end is notified of the new device. Subsequent communications will use this new bundle.
If an incoming message (on the same Whisper/Waku topic) fails to decrypt, a message is replied with the current bundle, so that the other end is notified of the new device. Subsequent communications will use this new bundle.
## Messaging
@ -97,11 +99,11 @@ The rest of this document is purely about 1:1 and private group chat. Private gr
### End-to-end encryption
End-to-end encryption (E2EE) takes place between two clients. The main cryptographic protocol is a [Status implementation](https://github.com/status-im/doubleratchet/) of the Double Ratchet protocol, which is in turn derived from the [Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html), using a different ratchet. The message payload is subsequently encrypted by the transport protocol - Whisper (see section [Transport Layer](#transport-layer)) -, using symmetric key encryption.
End-to-end encryption (E2EE) takes place between two clients. The main cryptographic protocol is a [Status implementation](https://github.com/status-im/doubleratchet/) of the Double Ratchet protocol, which is in turn derived from the [Off-the-Record protocol](https://otr.cypherpunks.ca/Protocol-v3-4.1.1.html), using a different ratchet. The message payload is subsequently encrypted by the transport protocol - Whisper/Waku (see section [Transport Layer](#transport-layer)) -, using symmetric key encryption.
Furthermore, Status uses the concept of prekeys (through the use of [X3DH](https://signal.org/docs/specifications/x3dh/)) to allow the protocol to operate in an asynchronous environment. It is not necessary for two parties to be online at the same time to initiate an encrypted conversation.
Status uses the following cryptographic primitives:
- Whisper
- Whisper/Waku
- AES-256-GCM
- ECIES
- ECDSA
@ -143,11 +145,11 @@ In the X3DH specification, a shared server is typically used to store bundles an
- QR codes;
- ENS record;
- Decentralized permanent storage (e.g. Swarm, IPFS).
- Whisper
- Whisper/Waku
<!-- TODO: Comment, it isn't clear what we actually _do_. It seems as if this is exploring the problem space. From a protocol point of view, it might make sense to describe the interface, and then have a recommendation section later on that specifies what we do. See e.g. Signal's specs where they specify specifics later on. -->
Currently only public and one-to-one message exchanges and Whisper is used to exchange bundles.
Currently only public and one-to-one message exchanges and Whisper/Waku is used to exchange bundles.
Since bundles stored in QR codes or ENS records cannot be updated to delete already used keys, the approach taken is to rotate more frequently the bundle (once every 24 hours), which will be propagated by the app through the channel available.
@ -252,7 +254,7 @@ message DirectMessageProtocol {
- `key`: Alice's ephemeral key `EK_A`;
- `id`: Identifier stating which of Bob's prekeys Alice used, in this case Bob's bundle signed prekey.
Alice's identity key `IK_A` is sent at the transport layer level (Whisper);
Alice's identity key `IK_A` is sent at the transport layer level (Whisper/Waku);
- `DR_header`: Double ratchet header ([protobuf](https://github.com/status-im/status-go/blob/a904d9325e76f18f54d59efc099b63293d3dcad3/services/shhext/chat/encryption.proto#L31)). Used when Bob's public bundle is available:
``` protobuf
@ -383,7 +385,7 @@ TODO: Verify if this can be done already by looking at Lamport clock difference
> Given a conversation transcript and all cryptographic keys, there is no evidence that a given message was authored by any particular user
- All messages are digitally signed by their sender.
- The underlying transport, Whisper, does allow for unsigned messages, but we don't use it.
- The underlying transport, Whisper/Waku, does allow for unsigned messages, but we don't use it.
#### Participant Repudiation (NO)
> Given a conversation transcript and all cryptographic key material for all but one accused (honest) participant, there is no evidence that the honest participant was in a conversation with any of the other participants.
@ -445,7 +447,7 @@ TODO: create issues for identity leak of invited members as well as current memb
- Due to asynchronous forward secrecy and no additional services, private keys might be rotated
[Andrea: That's correct, in some cases if the message is delayed for too long, or really out-of-order, the specific message key might have been deleted, as we only keep the last 3000 message keys]
[Igor: TTL of a whisper message can expire, so any node-in-transit will drop it. Also, I believe we ignore messages with skewed timestamps]
[Igor: TTL of a Whisper message can expire, so any node-in-transit will drop it. Also, I believe we ignore messages with skewed timestamps]
#### Dropped Message Resilient (PARTIAL)
> Messages can be decrypted without receipt of all previous messages. This is desirable for asynchronous and unreliable network services
@ -471,8 +473,8 @@ TODO: this requires more detail
#### No Additional Service (NO)
> The protocol does not require any infrastructure other than the protocol participants. Specifically, the protocol must not require additional servers for relaying messages or storing any kind of key material.
- The protocol requires whisper relay servers and mailservers currently.
- The larger the number of whisper relay servers, the better the transport security but there might be potential scaling problems.
- The protocol requires Whisper/Waku relay servers and mailservers currently.
- The larger the number of Whisper/Waku relay servers, the better the transport security but there might be potential scaling problems.
- Mailservers act to provide asynchronicity so users can retrieve messages after coming back from an offline period.
-->
@ -483,7 +485,7 @@ TODO: this requires more detail
A peer is identified by two pieces of data:
1) An `installation-id` which is generated upon creating a new account in the `Status` application
2) Their identity whisper key
2) Their identity Whisper/Waku key
### Initialization
@ -499,7 +501,7 @@ On receiving a bundle from a given peer with a higher version, the old bundle SH
### Multi-device support
Multi-device support is quite challenging as we don't have a central place where information on which and how many devices (identified by their respective `installation-id`) belongs to a whisper-identity.
Multi-device support is quite challenging as we don't have a central place where information on which and how many devices (identified by their respective `installation-id`) belongs to a whisper-identity / waku-identity.
Furthermore we always need to take account recovery in consideration, where the whole device is wiped clean and all the information about any previous sessions is lost.
@ -536,3 +538,11 @@ Account recovery is no different from adding a new device, and it is handled in
In some cases (i.e. account recovery when no other pairing device is available, device not paired), it is possible that a device will receive a message that is not targeted to its own `installation-id`.
In this case an empty message containing bundle information is sent back, which will notify the receiving end of including this device in any further communication.
## Changelog
### Version 0.3
Released `TODO`
- Added language to include Waku in all relevant places

24
docs/stable/6-payloads.md
View File

@ -6,7 +6,7 @@ title: 6/PAYLOADS
# 6/PAYLOADS
> Version: 0.2
> Version: 0.3
>
> Status: Stable
>
@ -50,6 +50,8 @@ as various clients created using different technologies.
- [Upgradability](#upgradability)
- [Security Considerations](#security-considerations)
- [Design rationale](#design-rationale)
- [Changelog](#changelog)
- [Version 0.3](#version-03)
## Introduction
@ -69,7 +71,7 @@ message StatusProtocolMessage {
`signature` is the bytes of the signed `SHA3-256` of the payload, signed with the key of the author of the message.
The signature is needed to validate authorship of the message, so that the message can be relayed to third parties.
If a signature is not present but an author is provided by a layer below, the message is not to be relayed to third parties and it is considered plausibly deniable.
If a signature is not present, but an author is provided by a layer below, the message is not to be relayed to third parties and it is considered plausibly deniable.
## Encoding
@ -89,7 +91,7 @@ The protobuf description is:
message ChatMessage {
// Lamport timestamp of the chat message
uint64 clock = 1;
// Unix timestamps in milliseconds, currently not used as we use whisper as more reliable, but here
// Unix timestamps in milliseconds, currently not used as we use Whisper/Waku as more reliable, but here
// so that we don't rely on it
uint64 timestamp = 2;
// Text of the message
@ -174,9 +176,9 @@ message StickerMessage {
#### Message types
Message types are required to decide how a particular message is encrypted and what metadata needs to be attached when passing a message to the transport layer. For more on this, see [3/WHISPER-USAGE](https://specs.status.im/spec/3).
Message types are required to decide how a particular message is encrypted and what metadata needs to be attached when passing a message to the transport layer. For more on this, see [3/WHISPER-USAGE](3-whisper-usage.md) and [9/WAKU-USAGE](9-waku-usage.md).
<!-- TODO: This reference is a bit odd, considering the layer payloads should interact with is Secure Transport, and not Whisper. This requires more detail -->
<!-- TODO: This reference is a bit odd, considering the layer payloads should interact with is Secure Transport, and not Whisper/Waku. This requires more detail -->
The following messages types MUST be supported:
@ -199,9 +201,9 @@ This will satisfy the Lamport requirement, namely: a -> b then T(a) < T(b)
`clock` SHOULD be calculated using the algorithm of [Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps). When there are messages available in a chat, `clock`'s value is calculated based on the last received message in a particular chat: `max(timeNowInMs, last-message-clock-value + 1)`. If there are no messages, `clock` is initialized with `timestamp`'s value.
Messages with a `clock` greater than `120` seconds over the whisper timestamp SHOULD be discarded, in order to avoid malicious users to increase the `clock` of a chat arbitrarily.
Messages with a `clock` greater than `120` seconds over the Whisper/Waku timestamp SHOULD be discarded, in order to avoid malicious users to increase the `clock` of a chat arbitrarily.
Messages with a `clock` less than `120` seconds under the whisper timestamp might indicate an attempt to insert messages in the chat history which is not distinguishable from a `datasync` layer re-transit event. A client MAY mark this messages with a warning to the user, or discard them.
Messages with a `clock` less than `120` seconds under the Whisper/Waku timestamp might indicate an attempt to insert messages in the chat history which is not distinguishable from a `datasync` layer re-transit event. A client MAY mark this messages with a warning to the user, or discard them.
`clock` value is used for the message ordering. Due to the used algorithm and distributed nature of the system, we achieve casual ordering which might produce counter-intuitive results in some edge cases. For example, when one joins a public chat and sends a message before receiving the exist messages, their message `clock` value might be lower and the message will end up in the past when the historical messages are fetched.
@ -332,3 +334,11 @@ There are two ways to upgrade the protocol without breaking compatibility:
-
## Design rationale
## Changelog
### Version 0.3
Released `TODO`
- Added language to include Waku in all relevant places

363
docs/stable/9-waku-usage.md Normal file
View File

@ -0,0 +1,363 @@
---
permalink: /spec/9
parent: Stable specs
title: 9/WAKU-USAGE
---
# 9/WAKU-USAGE
> Version: 0.1
>
> Status: Stable
>
> Authors: Adam Babik <adam@status.im>, Corey Petty <corey@status.im>, Oskar Thorén <oskar@status.im>, Samuel Hawksby-Robinson <samuel@status.im> (alphabetical order)
- [Status Waku Usage Specification](#9waku-usage)
- [Abstract](#abstract)
- [Reason](#reason)
- [Terminology](#terminology)
- [Waku packets](#waku-packets)
- [Waku node configuration](#waku-node-configuration)
- [Status](#status)
- [Rate limiting](#rate-limiting)
- [Keys management](#keys-management)
- [Contact code topic](#contact-code-topic)
- [Partitioned topic](#partitioned-topic)
- [Public chats](#public-chats)
- [Group chat topic](#group-chat-topic)
- [Negotiated topic](#negotiated-topic)
- [Message encryption](#message-encryption)
- [Message confirmations](#message-confirmations)
- [Waku V1 extensions](#waku-v1-extensions)
- [Request historic messages](#request-historic-messages)
- [wakuext_requestMessages](#wakuext_requestmessages)
- [Changelog](#changelog)
- [Version 0.1](#version-01)
## Abstract
Status uses [Waku](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md) to provide
privacy-preserving routing and messaging on top of devP2P. Waku uses topics
to partition its messages, and these are leveraged for all chat capabilities. In
the case of public chats, the channel name maps directly to its Waku topic.
This allows anyone to listen on a single channel.
Additionally, since anyone can receive Waku envelopes, it relies on the
ability to decrypt messages to decide who is the correct recipient. We do
however not rely on this property, but instead implement another secure
transport layer on top of Waku.
## Reason
Provide routing, metadata protection, topic-based multicasting and basic
encryption properties to support asynchronous chat.
## Terminology
* *Waku node*: an Ethereum node with Waku V1 enabled
* *Waku network*: a group of Waku nodes connected together through the internet connection and forming a graph
* *Message*: decrypted Waku message
* *Offline message*: an archived envelope
* *Envelope*: encrypted message with metadata like topic and Time-To-Live
## Waku packets
| Packet Name | Code | References |
| -------------------- | ---: | --- |
| Status | 0 | [Status](#status), [WAKU-1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#status) |
| Messages | 1 | [WAKU-1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#messages) |
| Batch Ack | 11 | Undocumented. Marked for Deprecation |
| Message Response | 12 | [WAKU-1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#message-confirmations-update) |
| Status Update | 22 | [WAKU-1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#status-update) |
| P2P Request Complete | 125 | [4/WAKU-MAILSERVER](https://specs.status.im/spec/4) |
| P2P Request | 126 | [4/WAKU-MAILSERVER](https://specs.status.im/spec/4), [WAKU-1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#p2p-request) |
| P2P Messages | 127 | [4/WAKU-MAILSERVER](https://specs.status.im/spec/4), [WAKU-1](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#p2p-message) |
## Waku node configuration
If you want to run a Waku node and receive messages from Status clients, it must be properly configured.
Waku's Proof Of Work algorithm is used to deter denial of service and various spam/flood attacks against the Waku network. The sender of a message must perform some work which in this case means processing time. Because Status' main client is a mobile client, this easily leads to battery draining and poor performance of the app itself. Hence, all clients MUST use the following Waku node settings:
* proof-of-work requirement not larger than `0.002` for payloads less than 50,000 bytes
* proof-of-work requirement not larger than `0.000002` for payloads greater than or equal to 50,000 bytes
* time-to-live not lower than `10` (in seconds)
## Status
Handshake is a RLP-encoded packet sent to a newly connected peer. It MUST start with a Status Code (`0x00`) and follow up with items:
```
[
[ pow-requirement-key pow-requirement ]
[ bloom-filter-key bloom-filter ]
[ light-node-key light-node ]
[ confirmations-enabled-key confirmations-enabled ]
[ rate-limits-key rate-limits ]
[ topic-interest-key topic-interest ]
]
```
| Option Name | Key | Type | Description | References |
| ----------------------- | ------ | -------- | ----------- | --- |
| `pow-requirement` | `0x00` | `uint64` | minimum PoW accepted by the peer | [WAKU-1#pow-requirement](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#pow-requirement-field) |
| `bloom-filter` | `0x01` | `[]byte` | bloom filter of Waku topic accepted by the peer | [WAKU-1#bloom-filter](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#bloom-filter-field) |
| `light-node` | `0x02` | `bool` | when true, the peer won't forward envelopes through the Messages packet. | `TODO` |
| `confirmations-enabled` | `0x03` | `bool` | when true, the peer will send message confirmations | `TODO` |
| `rate-limits` | `0x04` | | See [Rate limiting](#rate-limiting) | [WAKU-1#rate-limits](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#rate-limits-field) |
| `topic-interest` | `0x05` | `[10000][4]byte` | Topic interest is used to share a node's interest in envelopes with specific topics. It does this in a more bandwidth considerate way, at the expense of some metadata protection. Peers MUST only send envelopes with specified topics. | [WAKU-1#topic-interest](https://github.com/vacp2p/specs/blob/master/specs/waku/waku-1.md#topic-interest-field), [the theoretical scaling model](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability) |
<!-- TODO Add `light-node` and `confirmations-enabled` links when https://github.com/vacp2p/specs/pull/128 is merged -->
## Rate limiting
In order to provide an optional very basic Denial-of-Service attack protection, each node SHOULD define its own rate limits. The rate limits SHOULD be applied on IPs, peer IDs, and envelope topics.
Each node MAY decide to whitelist, i.e. do not rate limit, selected IPs or peer IDs.
If a peer exceeds node's rate limits, the connection between them MAY be dropped.
Each node SHOULD broadcast its rate limits to its peers using `rate limits` in `status-options` via packet code `0x00` or `0x22`. The rate limits is RLP-encoded information:
```
[ IP limits, PeerID limits, Topic limits ]
```
`IP limits`: 4-byte wide unsigned integer
`PeerID limits`: 4-byte wide unsigned integer
`Topic limits`: 4-byte wide unsigned integer
The rate limits MAY also be sent as an optional parameter in the handshake.
Each node SHOULD respect rate limits advertised by its peers. The number of packets SHOULD be throttled in order not to exceed peer's rate limits. If the limit gets exceeded, the connection MAY be dropped by the peer.
## Keys management
The protocol requires a key (symmetric or asymmetric) for the following actions:
* signing & verifying messages (asymmetric key)
* encrypting & decrypting messages (asymmetric or symmetric key).
As asymmetric keys and symmetric keys are required to process incoming messages,
they must be available all the time and are stored in memory.
Keys management for PFS is described in [5/SECURE-TRANSPORT](https://specs.status.im/spec/5).
The Status protocols uses a few particular Waku topics to achieve its goals.
### Contact code topic
Contact code topic is used to facilitate the discovery of X3DH bundles so that the first message can be PFS-encrypted.
Each user publishes periodically to this topic. If user A wants to contact user B, she SHOULD look for their bundle on this contact code topic.
Contact code topic MUST be created following the algorithm below:
```golang
contactCode := "0x" + hexEncode(activePublicKey) + "-contact-code"
var hash []byte = keccak256(name)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Partitioned topic
Waku is broadcast-based protocol. In theory, everyone could communicate using a single topic but that would be extremely inefficient. Opposite would be using a unique topic for each conversation, however, this brings privacy concerns because it would be much easier to detect whether and when two parties have an active conversation.
Partitioned topics are used to broadcast private messages efficiently. By selecting a number of topic, it is possible to balance efficiency and privacy.
Currently, the number of partitioned topics is set to `5000`. They MUST be generated following the algorithm below:
```golang
var partitionsNum *big.Int = big.NewInt(5000)
var partition *big.Int = big.NewInt(0).Mod(publicKey.X, partitionsNum)
partitionTopic := "contact-discovery-" + strconv.FormatInt(partition.Int64(), 10)
var hash []byte = keccak256(partitionTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Public chats
A public chat MUST use a topic derived from a public chat name following the algorithm below:
```golang
var hash []byte
hash = keccak256(name)
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
<!-- NOTE: commented out as it is currently not used. In code for potential future use. - C.P. Oct 8, 2019
### Personal discovery topic
Personal discovery topic is used to ???
A client MUST implement it following the algorithm below:
```golang
personalDiscoveryTopic := "contact-discovery-" + hexEncode(publicKey)
var hash []byte = keccak256(personalDiscoveryTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
Each Status Client SHOULD listen to this topic in order to receive ??? -->
### Group chat topic
Group chats does not have a dedicated topic. All group chat messages (including membership updates) are sent as one-to-one messages to multiple recipients.
### Negotiated topic
When a client sends a one to one message to another client, it MUST listen to their negotiated topic. This is computed by generating
a diffie-hellman key exchange between two members and taking the first four bytes of the `SHA3-256` of the key generated.
```golang
sharedKey, err := ecies.ImportECDSA(myPrivateKey).GenerateShared(
ecies.ImportECDSAPublic(theirPublicKey),
16,
16,
)
hexEncodedKey := hex.EncodeToString(sharedKey)
var hash []byte = keccak256(hexEncodedKey)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
A client SHOULD send to the negotiated topic only if it has received a message from all the devices included in the conversation.
### Flow
To exchange messages with client B, a client A SHOULD:
- Listen to client's B Contact Code Topic to retrieve their bundle information, including a list of active devices
- Send a message on client's B partitioned topic
- Listen to the Negotiated Topic between A & B
- Once a message is received from B, the Negotiated Topic SHOULD be used
## Message encryption
Even though, the protocol specifies an encryption layer that encrypts messages before passing them to the transport layer, Waku protocol requires each Waku message to be encrypted anyway.
Public and group messages are encrypted using symmetric encryption and the key is created from a channel name string. The implementation is available in [`waku_generateSymKeyFromPassword`](https://github.com/status-im/status-go/tree/develop/_examples) JSON-RPC method of go-ethereum Whisper implementation.
One-to-one messages are encrypted using asymmetric encryption.
## Message confirmations
Sending a message is a complex process where many things can go wrong. Message confirmations tell a node that a message originating from it has been seen by its direct peers.
A node MAY send a message confirmation for any batch of messages received in a packet Messages Code (`0x01`).
A message confirmation is sent using Batch Acknowledge packet (`0x0b`) or Message Response packet (`0x0c`).
The Batch Acknowledge packet is followed by a keccak256 hash of the envelopes batch data (raw bytes).
The Message Response packet is more complex and is followed by a Versioned Message Response:
```
[ Version, Response]
```
`Version`: a version of the Message Response, equal to `1`,
`Response`: `[ Hash, Errors ]` where `Hash` is a keccak256 hash of the envelopes batch data (raw bytes) for which the confirmation is sent and `Errors` is a list of envelope errors when processing the batch. A single error contains `[ Hash, Code, Description ]` where `Hash` is a hash of the processed envelope, `Code` is an error code and `Description` is a descriptive error message.
The supported codes:
`1`: means time sync error which happens when an envelope is too old or created in the future (the root cause is no time sync between nodes).
The drawback of sending message confirmations is that it increases the noise in the network because for each sent message, a corresponding confirmation is broadcast by one or more peers. To limit that, both Batch Acknowledge packet (`0x0b`) and Message Response packet (`0x0c`) are not broadcast to peers of the peers, i.e. they do not follow epidemic spread.
In the current Status network setup, only `Mailservers` support message confirmations. A client posting a message to the network and after receiving a confirmation can be sure that the message got processed by the `Mailserver`. If additionally, sending a message is limited to non-`Mailserver` peers, it also guarantees that the message got broadcast through the network and it reached the selected `Mailserver`.
## Waku V1 extensions
### Request historic messages
Sends a request for historic messages to a `Mailserver`. The `Mailserver` node MUST be a direct peer and MUST be marked as trusted (using `waku_markTrustedPeer`).
The request does not wait for the response. It merely sends a peer-to-peer message to the `Mailserver` and it's up to `Mailserver` to process it and start sending historic messages.
The drawback of this approach is that it is impossible to tell which historic messages are the result of which request.
It's recommended to return messages from newest to oldest. To move further back in time, use `cursor` and `limit`.
#### wakuext_requestMessages
**Parameters**:
1. Object - The message request object:
* `mailServerPeer` - `String`: `Mailserver`'s enode address.
* `from` - `Number` (optional): Lower bound of time range as unix timestamp, default is 24 hours back from now.
* `to` - `Number` (optional): Upper bound of time range as unix timestamp, default is now.
* `limit` - `Number` (optional): Limit the number of messages sent back, default is no limit.
* `cursor` - `String` (optional): Used for paginated requests.
* `topics` - `Array`: hex-encoded message topics.
* `symKeyID` - `String`: an ID of a symmetric key used to authenticate with the `Mailserver`, derived from the `Mailserver` password.
**Returns**:
`Boolean` - returns `true` if the request was sent.
The above `topics` is then converted into a bloom filter and then and sent to the `Mailserver`.
<!-- TODO: Clarify actual request with bloom filter to mailserver -->
## Changelog
### Version 0.1
Released `TODO`
- Created document
- Forked from [3-whisper-usage](3-whisper-usage.md)
- Change to keep `Mailserver` term consistent
- Replaced Whisper references with Waku
- Added [Status options](#status) section
- Updated [Waku packets](#waku-packets) section to match Waku
- Added that `Batch Ack` is marked for deprecation
- Changed `shh_generateSymKeyFromPassword` to `waku_generateSymKeyFromPassword`
- [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/waku/api.go#L172-L175)
- [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/eth-node/bridge/geth/public_waku_api.go#L33-L36)
- Changed `shh_markTrustedPeer` to `waku_markTrustedPeer`
- [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/waku/api.go#L100-L108)
- Changed `shhext_requestMessages` to `wakuext_requestMessages`
- [Exists here](https://github.com/status-im/status-go/blob/2d13ccf5ec3db7e48d7a96a7954be57edb96f12f/services/wakuext/api.go#L76-L139)

5
wordlist.txt
View File

@ -78,6 +78,7 @@ Gheorghe
Gmbh
golang
Guilherme
Hawksby
HD
HeaderByNumber
hellman
@ -194,6 +195,7 @@ topicLen
TransactionByHash
TransactionReceipt
trilemma
TTL
TXT
UI
uint
@ -208,6 +210,9 @@ UX
Volodymyr
Vp
Waku
Waku's
waku
wakuext
wei
Whitepaper
wordlist