status-im
/
specs
mirror of https://github.com/status-im/specs.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #17 from status-im/status-protocol-overview 

edit Initial Status Protocol Overview (closes #11)
This commit is contained in:
Oskar Thorén 2019-07-12 10:41:28 +08:00 committed by GitHub
parent ca84357ac7 1aa1860ab2
commit 77535ff759
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 49 additions and 336 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

385
x4.md
View File

@ -1,378 +1,91 @@
---
sip: x4
title: Initial Protocol Overview
title: Initial Status Protocol Overview
status: Draft
type: Standard
author: Adam Babik <adam@status.im>, Oskar Thorén <oskar@status.im>
author: Adam Babik <adam@status.im>, Oskar Thorén <oskar@status.im>
created: 2019-04-18
updated:
---
**TODO: Add special considerations when it comes to putting these together**
**TODO: Remove everything that's Whisper specific, conversational security, payloads, etc.**
# Protocol Overview
The Status protocol stack consists of multiple protocols at multiple levels. In this document, we give a brief overview of each layer and what they provide. We also note any specific considerations when it comes to using these components together.
| Layer | Purpose | Technology |
|-------------------|---------------------------------|-----------------------|
| Message payloads | End user functionality | 1:1, group chat, public chat |
| Secure Transport | Confidentiality, PFS, etc | Double Ratchet |
| Transport Privacy | Metadata protection | Whisper |
| P2P Overlay | Overlay routing, NAT traversal | devp2p |
Note that our protocol stack was documented after the fact, which might lead to some inconsistencies or questionable design choices.
## Guiding principles
**TODO: This needs some tweaking, as it was taken from https://github.com/status-im/swarms/blob/master/ideas/311-status-protocol.md which also includes how to work, as opposed to how the prinicples apply technically**
One way of looking at our protocol stack is as a reflection of our principles. While this isn't always true in practice right now, this is what we are aiming for, and also what we want to see in future specification proposals.
The protocol should be a reflection of our principles.
I. Liberty - enable sovereignty of individuals with things like key management, as well as economic freedom, etc, by enabling transaction of funds and socio-economic coordination between small groups of people. This also includes being coercion resistant, and not forcing updates on people.
II. Censorship resistance - enable censorship resistance through things like pluggable transports, as well as being agnostic to the information being transported.
III. Security - use state-of-the-art technologies, and research new security methods and technologies to make strong security guarantees.
IV. Privacy - protect privacy in communication and transactions, and strive to provide right to total anonymity.
V. Transparency - be open about what we are doing, allow community contributions and be clear about trade-offs.
VI. Openness - specification is open and under a permissive license.
VII. Decentralization - p2p network, maximizing number of computers/humans who can control and use the protocol we are building.
VIII. Inclusivity - easy to use our protocols, as well as working towards interopability with other protocols, and educating users.
IX. Continuance - network should be incentivized to continue on its own.
X. Resourcefulness - work with other teams to avoid duplication of effort, study prior work deeply to minimize wasted and duplicate effort.
In the following sections we will elaborate more on these.
# Status Secure Messaging Protocol
**TODO: Didn't get this far yet -- Oskar**
- [Status Secure Messaging Protocol](#status-secure-messaging-protocol)
- [Abstract](#abstract)
- [Terminology](#terminology)
- [Basic Assumption](#basic-assumption)
- [Protocol Overview](#protocol-overview)
- [Payload](#payload)
- [Content types](#content-types)
- [Message types](#message-types)
- [Clock vs Timestamp and message ordering](#clock-vs-timestamp-and-message-ordering)
- [Replies](#replies)
- [Whisper adapter](#whisper-adapter)
- [Whisper node configuration](#whisper-node-configuration)
- [Keys management](#keys-management)
- [Encryption algorithms](#encryption-algorithms)
- [Topic](#topic)
- [Message encryption](#message-encryption)
- [Perfect Forward Secrecy (PFS)](#perfect-forward-secrecy-pfs)
- [Device syncing](#device-syncing)
- [One-to-one messages](#one-to-one-messages)
- [Sending](#sending)
- [Sending using PFS](#sending-using-pfs)
- [Receiving](#receiving)
- [Public messages](#public-messages)
- [Sending](#sending-1)
- [Receiving](#receiving-1)
- [Group messages](#group-messages)
- [Offline messages](#offline-messages)
- [Anonymity concerns](#anonymity-concerns)
- [Whisper V6 extensions (or Status Whisper Node)](#whisper-v6-extensions-or-status-whisper-node)
- [New RPC methods](#new-rpc-methods)
# Abstract
Ethereum empowers users and developers to interact with totally new kind of applications called Dapps (Decentralized applications). These application allows to interact with the blockchain on a completely new level which is not only about exchanging values but also executing arbitrary logic. This logic can form very sophisticated programs like DAOs (Decentralized autonomous organizations). The missing part here is how users of Dapps can communicate securely and in a decentralized way with each other. Communication is an essential part of any activity. In this document, we specify a secure and decentralized messaging protocol that is capable of running on the Ethereum network.
Ethereum empowers users and developers to interact with totally new kind of applications called Dapps (Decentralized applications). These application allows to interact with the blockchain on a completely new level which is not only about exchanging values but also executing arbitrary logic. This logic can form very sophisticated programs like DAOs (Decentralized autonomous organizations). The missing part here is how users of Dapps can communicate securely and in a decentralized way with each other. Communication is an essential part of any activity. In this document, we specify a secure and decentralized messaging protocol, onwards called Status Protocol, that is capable of running on the Ethereum network.
# Protocol overview
# Terminology
TODO:
* why p2p?
* why on Ethereum stack?
* designed for mobile from the beginning
**TODO: Most of these can probably be moved to transport privacy through Whisper spec**
# Protocol Layers
* *Client*: a Whisper node implementing the protocol
* *Whisper node*: an Ethereum node with Whisper V6 enabled (in the case of geth, it's `--shh` option)
* *Status Whisper node*: an Ethereum node with Whisper V6 enabled and additional Whisper extensions described in [Whisper V6 extensions (or Status Whisper Node)](#whisper-v6-extensions-or-status-whisper-node)
* *Whisper network*: a group of Whisper nodes connected together through the internet connection and forming a graph
* *MailServer*: an Ethereum node with Whisper V6 enabled and a mail server registered capable of storing and providing offline messages
* *Message*: decrypted Whisper message
* *Envelope*: encrypted message with some metadata like topic and TTL sent between Whisper nodes; a symmetric or asymmetric key is needed to decrypt it and read the payload
* *Offline message*: an expired envelope stored by a Whisper node permanently
The Status Protocol stack consists of multiple protocols. In this document, we give a brief overview of each layer and what they provide. We also note any specific considerations when it comes to using these components together.
# Basic Assumption
| Layer | Purpose | Technology |
|-------------------|---------------------------------|------------------------------|
| Data | End user functionality | 1:1, group chat, public chat |
| Security | Confidentiality, PFS | Double Ratchet |
| Privacy | Metadata protection | Whisper |
| P2P | Overlay routing, NAT traversal | devp2p |
This protocol assumes the following:
1. There MUST be an Ethereum node that is capable of discovering peers and implements Whisper V6 specification.
2. Participants of a given Whisper network in order to communicate with each other MUST accept messages with lowered PoW value. More in (Whisper node configuration)(#whisper-node-configuration).
3. Time MUST be synced between all nodes participating in the given network (this is intrinsic requirement of the Whisper specification as well). A clock drift between two peers larger than 20 seconds MAY result in discarding incoming messages.
> Note that the Status Protocol was documented after the fact, which might lead to some inconsistencies or questionable design choices.
# Protocol Overview
# Data layer
Notice: this protocol is documented post factum. The goal of it is to clearly present the current design and prepare the ground for its second version.
Data layer specifies a format of payloads exchanged between clients communicating using the Status Protocol. The robust and well-defined format is required in order to provide some security guarantees, for example, client should not display messages that do not conform to the format, and allow the clients to upgrade the protocol safely in the future.
The implementation of this protocol is mainly done in https://github.com/status-im/status-react and https://github.com/status-im/status-go repositories.
The well-defined format also makes it possible to support multiple types of conversations as well as multiple types of content.
The goal of this protocol is to allow people running Ethereum nodes with Whisper service enabled to exchange messages that are end-to-end encrypted in a way that guarantees [darkness to some extent](https://github.com/ethereum/go-ethereum/wiki/Achieving-Darkness).
Please refer to [Initial Message Payload Specification](x8.md) for more details.
It's important to notice that messages [are not limited to be text messages](#content-types) only. They can also have special meaning depending on the client implementation. For example, in the current implementation, there are message which informs about Eth requests.
# Conversational Security Layer
This protocol consist of three components:
* [payload](x8.md)
* Whisper adapter
* offline messaging.
Conversational Security Layer provides various cryptographical properties to the Status Protocol:
* confidentiality
* integrity
* authentication
* forward secrecy.
[The payload section](#payload) describes how the messages are encoded and decoded and what each fields of a message means. This is required to properly interpret messages by the client.
Confidetiality makes sure that only intended recipients are able to read a message.
Whisper adapter specifies interaction with the Whisper service with regards to keys management, configuration and attaching metadata required to properly forward and process messages.
Integrity means that no honest party will accept a message modified in the transit.
Offline messaging describes how the protocol handles delivering messages when one or more participants are offline and the messages expire in the Whisper network.
Authentication means that each participant in the conversation receives a proof of possession of a known long-term secret from all other participants. In addition, each participant is able to verify that a message was sent from the claimed source.
The protocol does not specify additional things like peers discovery, running Whisper nodes, underlying p2p protocols etc.
This assumes trust has already been established. Initial trust establishment is detailed in [Initial Trust Establishment Specification](x5.md).
Forward secrecy, known also as perfect forwardsecrecy (PFS), gives assurance that session keys will not be compromised even if the private key is compromised. Also, compromising one session key will not result in compromising other sessions.
# Whisper adapter
Note: The Status Procol can work with PFS enabled and disabled. In the case of disabled PFS, the encryption is moved to the Privacy Layer and handled by Whisper. Whisper does **not** provide forward secrecy. This is a flaw of the original design which mixes security and private layer responsibilities. With PFS enabled, a message is encrypted twice.
Whisper in version 6 has been chosen as an messages exchange protocol because it was designed as an off-chain communication layer for the Ethereum nodes. It supports e2e encryption and uses epidemic spread to route data to all members of the network. It also provides [darkness to some extent](https://github.com/ethereum/go-ethereum/wiki/Achieving-Darkness).
Please refer to [Initial Conversational Security Specification](x6.md) for more details.
However, using Whisper has a few tradeoffs:
* was not designed to handle huge number of messages
* was not designed to be real-time; some delays over a few seconods are expected
* does not scale well with the number of messages in the network
# Transport privacy layer
This protocol can operate using a Whisper service which requires this protocol implementation to run in the same process as well as Whisper's RPC API which might be provided by a separate Whisper node process via IPC or WebSocket.
The purpose of the transport privacy layer is to hide metadata (provide darkness) that are always sent when communicating over the internet. In some cases, leaked metadata might be as dangerous as leaking the actual payload so it's important to reduce that.
There is some tight coupling between the payload and Whisper:
* Whisper message topic depends on the actual message type (see [Topic](#topic))
* Whisper message uses a different key (asymmetric or symmetric) depending on the actual message type (see [Keys management](#keys-management))
The Status Protocol uses [Whisper in version 6](https://github.com/ethereum/wiki/wiki/Whisper) as a privacy protection protocol. Whisper routes messages by broadcasting them to all connected peers so that identifying of the recipient is almost impossible (the sender is not equally protected, though). [Read more](https://github.com/ethereum/go-ethereum/wiki/Achieving-Darkness) about how Whisper achieves darkness.
## Whisper node configuration
Note: there is one more cross-layer dependency in the Status Protocol between the privacy and data layers. Whisper uses topics to reduce traffic and preliminarily filter out messages. However, selecting a topic depends on the type of the message that is a topic for 1-1 chat will be different from a topic for a public chat.
If you want to run a Whisper node and receive messages from Status clients, it must be properly cnofigured.
Please refer to [Initial Transport Privacy through Whisper Specification](x7.md) for more details.
Whisper's Proof Of Work algorithm is used to deter denial of service and various spam/flood attacks against the Whisper network. The sender of a message must perform some work which in this case means processing time. Because Status' main client is a mobile client, this easily leads to battery draining and poor performance of the app itself. Hence, all clients MUST use the following Whisper node settings:
* proof-of-work not larger than `0.002`
* time-to-live not lower than `10` (in seconds)
# P2P layer
TODO: provide an instruction how to start a Whisper node with proper configuration using geth.
P2P layer allows various clients to securely communicate with each other through the internet in a decetralized fashion, envetually forming a peer-to-peer network. Due to our close relationship to Ethereum and the fact that Whisper is built on top of [devp2p](https://github.com/ethereum/devp2p), we use [devp2p](https://github.com/ethereum/devp2p) as a P2P layer.
## Keys management
A client in order to rely a message needs to first find other clients (this process is called node discovery and is out of scope of this specification) within the peer-to-peer network. Then, the message needs to be properly routed handling obstacles like NAT traversal. All these things are handled by devp2p.
The protocol requires a key (symmetric or asymmetric) for the following actions:
* signing a message (a private key)
* decrypting received messages (a private key or symmetric key).
# Censorship resistance
As private keys and symmetric keys are required to process incoming messages, they must be available all the time and are stored in memory.
TODO
Keys management for PFS is described in [Perfect forward secrecy section](#perfect-forward-secrecy-pfs).
# Dependencies between layers
## Encryption algorithms
As it was mentioned above, there are cases when two layers depend on one another:
* encrypting messages is done by the privacy layer instead of security layer
* the privacy layer needs to know some details of the payload to properly route it.
All encryption algorithms used by Whisper should be described in the [Whisper V6 specification](http://eips.ethereum.org/EIPS/eip-627).
Cryptographic algoritms used by PFS are described in [Perfect forward secrecy section](#perfect-forward-secrecy-pfs).
## Topic
There are two types of Whisper topics the protocol uses:
* static topic for `user-message` message type (also called _contact discovery topic_)
* dynamic topic based on a chat name for `public-group-user-message` message type.
The static topic is always the same and its hex representation is `0xf8946aac`. In fact, _the contact discovery topic_ is calculated using a dynamic topic algorithm described below with a constant name `contact-discovery`.
Having only one topic for all private chats has an advantage as it's very hard to figure out who talks to who. A drawback is that everyone receives everyone's messages but they can decrypt only these they have private keys for.
A dynamic topic is derived from a string using the following algorithm:
```
var hash []byte
hash = keccak256(name)
# Whisper V6 specific
var topic [4]byte
topic_len = 4
if len(hash) < topic_len {
topic_len = len(hash)
}
for i = 0; i < topic_len; i++ {
topic[i] = hash[i]
}
```
## Message encryption
The protocol distinguishes messages encrypted using asymmetric and symmetric encryption.
Symmetric keys are created using [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) Whisper V6 RPC API method which accepts one param, a string.
Messages encrypted with asymmetric encryption should be encrypted using recipient's public key so that only the recipient can decrypt them.
Encryption of messages supporting PFS is described in [Perfect Forward Secrecy](#perfect-forward-secrecy-pfs) section.
# Perfect Forward Secrecy (PFS)
Additionally to encrypting messages on the Whisper level, the protocol supports PFS specification.
A message payload is first encrypted following the PFS specification and then it is encrypted once again following the Whisper specification and this protocol.
As not all messages are encrypted with PFS, a following strategy MAY be used:
1. First, message is decrypted on the Whisper level
2. Try to decrypt the message payload using PFS algorithm
2.1. If successful, pass the decrypted value to (3)
2.2. If failed, pass the unchanged payload to (3)
3. Decode the payload as described in [Paylooad](#payload) section
TODO: link to a separate document (currently in the PR).
[PFS in Status.im docs](https://status.im/research/pfs.html)
# Device syncing
TODO: link to a separate document.
# One-to-one messages
One-to-one messages are also known as private messages. These are the messages sent beween two participants of the conversation.
## Sending
Sending a message is fairly easy and relies on the Whisper RPC API, however, some preparation is needed:
1. Obtain a public key of the recipient of the message
2. Add your private key to Whisper using [`shh_addPrivateKey`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_addprivatekey) and save the result as `sigKeyID`
3. Call [`shh_post`(https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_post) with the following settings:
1. `pubKey` MUST be a hex-encoded public key of the message recipient
2. `sig` MUST be set to `sigKeyID`
3. `ttl` MUST be at least `10` (it is in seconds)
4. `topic` MUST be set accordingly to [Topic](#topic) section and hex-encoded
5. `payload` MUST be a hex-encoded string
6. `powTime` MAY be arbitrary but should be enough to perform proof-of-work
7. `powTarget` MUST be equal or lower than `0.002`.
Note: these instructions are for the Whisper V6 RPC API. If you use Whisper service directly or Go `shhclient`, the parameters might have different types.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
### Sending using PFS
When one decides to use PFS, the flow is the same but the payload MUST be additionally encrypted following the [PFS specification](#pfs) before being hex-endoded and passed to `shh_post`.
## Receiving
Receiving private messages depends on Whisper filters idea. Upon receiving, messages are first matched by a topic and then trying to be decrypted using user's private key.
1. Add your private key to Whisper using [`shh_addPrivateKey`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_addprivatekey) and save the result as `sigKeyID`
2. Call [`shh_subscribe`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_subscribe) with criteria:
1. `minPow` MUST be at least `0.002`
2. `topics` MUST be list of hex-encoded topics you expect messages to receive from (follow [Topic](#topic) section)
3. `allowP2P` MUST be set to `true` if offline messages are supported, otherwise can be `false`.
Alternative method is to use [`shh_newMessageFilter`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_newmessagefilter) which takes the same criteria object and then periodically calling [`shh_getFilterMessages`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_getfiltermessages) method.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
# Public messages
Public messages are encrypted with a symmetric key which is publicly known so anyone can participate in the conversation.
The fact that anyone can participate makes the public chats voulnerable to spam attacks. Also, there are no moderators of these chats.
## Sending
1. Calculate a symmetric key using [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) passing a public chat name as a string and save the result to `symKeyID`
2. Call [`shh_post`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_post) with the following settings:
1. `symKeyID` MUST be set to `symKeyID`
2. `sig` MUST be set to `sigKeyID`
3. `ttl` MUST be at least `10` (it is in seconds)
4. `topic` MUST be set accordingly to [Topic](#topic) section and hex-encoded,
5. `payload` MUST be a hex-encoded string,
6. `powTime` MAY be arbitrary but should be enough to perform proof-of-work
7. `powTarget` MUST be equal or lower than `0.002`.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
## Receiving
Receiving public messages depends on Whisper filters idea. Upon receiving, messages are first matched by a topic and then trying to be decrypted using a symmetric key.
1. Calculate a symmetric key using [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) passing public chat name as a string and save the result to `symKeyID`
2. Call [`shh_subscribe`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_subscribe) with criteria:
1. `minPow` MUST be at least `0.002`
2. `topics` MUST be list of hex-encoded topics you expect messages to receive from (follow [Topic](#topic) section)
3. `allowP2P` MUST be set to `true` if offline messages are supported, otherwise can be `false`.
Alternative method is to use [`shh_newMessageFilter`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_newmessagefilter) which takes the same criteria object and then periodically calling [`shh_getFilterMessages`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_getfiltermessages) method.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
# Group messages
TODO: describe how to send a group message starting from adding a key in Whisper etc.
# Offline messages
In the case of mobile clients which are often offline, there is a strong need to have an ability to download offline messages. By offline messages, we mean messages sent into the Whisper network and expired before being collected by the recipient. A message stays in the Whisper network for a duration specified as `TTL` (time-to-live) property.
A Whisper client needs to register a mail server instance which will be used by [geth's Whisper service](https://github.com/ethereum/go-ethereum/blob/v1.8.23/whisper/whisperv6/whisper.go#L209-L213).
`MailServer` is an interface with two methods:
* `Archive(env *Envelope)`
* `DeliverMail(whisperPeer *Peer, request *Envelope)`
If a mail server is registered for a given Whisper client, it will save all incoming messages on a local disk (this is the simplest implementation, it can store the messages wherever it wants, also using technologies like swarm and IPFS) in the background.
Notice that each node is meant to be independent and SHOULD keep a copy of all historic messages. High Availability (HA) can be achieved by having multiple nodes in different locations. Additionally, each node is free to store messages in a way which provides storage HA as well.
Saved messages are delivered to a requester (another Whisper peer) asynchronously as a response to `p2pMessageCode` message code. This is not exposed as a JSON-RPC method in `shh` namespace but it's exposed in status-go as `shhext_requestMessages` and blocking `shh_requestMessagesSync`. Read more about [Whisper V6 extensions](#whisper-v6-extensions-or-status-whisper-node).
In order to receive historic messages from a filter, p2p messages MUST be allowed when creating the filter. Receiving p2p messages is implemented in [geth's Whisper V6 implementation](https://github.com/ethereum/go-ethereum/blob/v1.8.23/whisper/whisperv6/whisper.go#L739-L751).
## Anonymity concerns
In order to use a mail server, a given node needs to connect to it directly, i.e. add the mail server as its peer and mark it as trusted. This means that the mail server is able to send direct p2p messages to the node instead of broadcasting them. Effectively, it knows which topics the node is interested in, when it is online as well as many metadata like IP address.
# Whisper V6 extensions (or Status Whisper Node)
Protocol's target is to be compliant with [the Whisper V6 Specification](https://github.com/ethereum/go-ethereum/wiki/Whisper). It should not matter which implementation is used as long as the implementation follow the Whisper V6 Specification.
However, we added a few extensions, message codes and RPC methods to the Whisper V6 service in order to provide better user experience or due to efficiency requirements.
All described addons are implemented in [status-im/whisper fork](https://github.com/status-im/whisper).
## New RPC methods
TODO: provide a list of RPC methods from `shhext` API which are relevant to this spec.
## Raw SoK TODO leftovers
## Misc notes
### Factors not captured
- Censorship-resistance, DPI etc
- Continuance of protocol
- Availability/Robustness e.g. FW port blocking
- p2p no central source of control
- trust minimized
- Open machine readable specification
- multiple implementaions
- group chat performance
- mobile specific / churnable
### TODO: Later, for each property have
- Current protocl
- How we use it
- Future considerations
- Desired by Status
- The ethereum txn viewpoint
In the next version of the protocol, we will try to remove these dependencies so that all layers are orthogonal and if necessary there is a clean interface between one and another.