status-im/specs
2
0
Fork 0
mirror of https://github.com/status-im/specs.git synced 2025-01-27 06:34:58 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #52 from status-im/cleanup-whisper-usage

Browse Source 
Clean up spec about Whisper usage
This commit is contained in:
Corey 2019-10-08 08:42:56 -04:00 committed by GitHub
commit 62e7eb0287
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 138 additions and 229 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

367
status-whisper-usage-spec.md
View File

@ -4,10 +4,28 @@
>
> Authors: Adam Babik <adam@status.im>, Corey Petty <corey@status.im>, Oskar Thorén <oskar@status.im> (alphabetical order)
- [Status Whisper Usage Specification](#status-whisper-usage-specification)
- [Abstract](#abstract)
- [Reason](#reason)
- [Terminology](#terminology)
- [Whisper node configuration](#whisper-node-configuration)
- [Keys management](#keys-management)
- [Contact code topic](#contact-code-topic)
- [Partitioned topic](#partitioned-topic)
- [Public chats](#public-chats)
- [Personal discovery topic](#personal-discovery-topic)
- [Generic discovery topic](#generic-discovery-topic)
- [One-to-one topic](#one-to-one-topic)
- [Group chat topic](#group-chat-topic)
- [Message encryption](#message-encryption)
- [Whisper V6 extensions](#whisper-v6-extensions)
- [Request historic messages](#request-historic-messages)
- [shhext_requestMessages](#shhextrequestmessages)
## Abstract
Status uses [Whisper](https://eips.ethereum.org/EIPS/eip-627) to provide
privacy-preserving routing and messaging on top of DevP2P. Whisper uses topics
privacy-preserving routing and messaging on top of devP2P. Whisper uses topics
to partition its messages, and these are leveraged for all chat capabilities. In
the case of public chats, the channel name maps directly to its Whisper topic.
This allows allows anyone to listen on a single channel.
@ -20,105 +38,18 @@ transport layer on top of Whisper.
Finally, we use an extension of Whisper to provide the ability to do offline
messaging.
## Table of Contents
- [Abstract](#abstract)
- [Table of Contents](#table-of-contents)
- [Introduction](#introduction)
- [Requirements](#requirements)
- [Design goals](#design-goals)
- [Terminology](#terminology)
- [Basic Assumption](#basic-assumption)
- [Protocol Overview](#protocol-overview)
- [Whisper adapter](#whisper-adapter)
- [Whisper node configuration](#whisper-node-configuration)
- [Keys management](#keys-management)
- [Encryption algorithms](#encryption-algorithms)
- [Topics](#topics)
- [Message encryption](#message-encryption)
- [Perfect Forward Secrecy (PFS)](#perfect-forward-secrecy-pfs)
- [Device syncing](#device-syncing)
- [One-to-one messages](#one-to-one-messages)
- [Sending](#sending)
- [Sending using PFS](#sending-using-pfs)
- [Receiving](#receiving)
- [Public messages](#public-messages)
- [Sending](#sending-1)
- [Receiving](#receiving-1)
- [Offline messages](#offline-messages)
- [Whisper V6 extensions (or Status Whisper Node)](#whisper-v6-extensions-or-status-whisper-node)
- [Security concerns](#security-concerns)
## Introduction
In this document we detail how we use Whisper to provide for the various chat
use cases, as well how offline inboxing works.
## Requirements
An Ethereum node that is connected to peers and implements the Whisper v6
specifications.
## Design goals
## Reason
Provide routing, metadata protection, topic-based multicasting and basic
encryption properties to support asynchronous chat.
## Terminology
* *Client*: a Whisper node implementing the protocol
* *Whisper node*: an Ethereum node with Whisper V6 enabled (in the case of geth, it's `--shh` option)
* *Status Whisper node*: an Ethereum node with Whisper V6 enabled and additional Whisper extensions described in [Whisper V6 extensions (or Status Whisper Node)](#whisper-v6-extensions-or-status-whisper-node)
* *Whisper network*: a group of Whisper nodes connected together through the internet connection and forming a graph
* *MailServer*: an Ethereum node with Whisper V6 enabled and a mail server registered capable of storing and providing offline messages
* *Message*: decrypted Whisper message
* *Envelope*: encrypted message with some metadata like topic and TTL sent between Whisper nodes; a symmetric or asymmetric key is needed to decrypt it and read the payload
* *Offline message*: an expired envelope stored by a Whisper node permanently
## Basic Assumption
This protocol assumes the following:
1. There MUST be an Ethereum node that is capable of discovering peers and implements Whisper V6 specification.
2. Participants of a given Whisper network in order to communicate with each other MUST accept messages with lowered PoW value. More in (Whisper node configuration)(#whisper-node-configuration).
3. Time MUST be synced between all nodes participating in the given network (this is intrinsic requirement of the Whisper specification as well). A clock drift between two peers larger than 20 seconds MAY result in discarding incoming messages.
## Protocol Overview
Notice: this protocol is documented post factum. The goal of it is to clearly present the current design and prepare the ground for its second version.
The implementation of this protocol is mainly done in https://github.com/status-im/status-react and https://github.com/status-im/status-go repositories.
The goal of this protocol is to allow people running Ethereum nodes with Whisper service enabled to exchange messages that are end-to-end encrypted in a way that guarantees [darkness to some extent](https://github.com/ethereum/go-ethereum/wiki/Achieving-Darkness).
It's important to notice that messages [are not limited to be text messages](#content-types) only. They can also have special meaning depending on the client implementation. For example, in the current implementation, there are message which informs about Eth requests.
This protocol consist of three components:
* payload
* Whisper adapter
* offline messaging.
[The payload section](#payload) describes how the messages are encoded and decoded and what each fields of a message means. This is required to properly interpret messages by the client.
Whisper adapter specifies interaction with the Whisper service with regards to keys management, configuration and attaching metadata required to properly forward and process messages.
Offline messaging describes how the protocol handles delivering messages when one or more participants are offline and the messages expire in the Whisper network. For more, see [Status Whisper Mailserver Spec)[status-whisper-mailserver-spec.md].
The protocol does not specify additional things like peers discovery, running Whisper nodes, underlying p2p protocols etc.
## Whisper adapter
Whisper in version 6 has been chosen as an messages exchange protocol because it was designed as an off-chain communication layer for the Ethereum nodes. It supports e2e encryption and uses epidemic spread to route data to all members of the network. It also provides [darkness to some extent](https://github.com/ethereum/go-ethereum/wiki/Achieving-Darkness).
However, using Whisper has a few tradeoffs:
* was not designed to handle huge number of messages
* was not designed to be real-time; some delays over a few seconods are expected
* does not scale well with the number of messages in the network
This protocol can operate using a Whisper service which requires this protocol implementation to run in the same process as well as Whisper's RPC API which might be provided by a separate Whisper node process via IPC or WebSocket.
There is some tight coupling between the payload and Whisper:
* Whisper message topic depends on the actual message type (see [Topic](#topic))
* Whisper message uses a different key (asymmetric or symmetric) depending on the actual message type (see [Keys management](#keys-management))
* *Offline message*: an archived envelope
* *Envelope*: encrypted message with metadata like topic and Time-To-Live
## Whisper node configuration
@ -143,167 +74,145 @@ they must be available all the time and are stored in memory.
Keys management for PFS is described in [Perfect forward secrecy section](#perfect-forward-secrecy-pfs).
## Encryption algorithms
The Status protocols uses a few particular Whisper topics to achieve its goals.
All encryption algorithms used by Whisper should be described in the [Whisper V6 specification](http://eips.ethereum.org/EIPS/eip-627).
### Contact code topic
Cryptographic algoritms used by PFS are described in [Perfect forward secrecy section](#perfect-forward-secrecy-pfs).
Contact code topic is used to facilitate the discovery of X3DH bundles so that the first message can be PFS-encrypted.
### Topics
There are two types of Whisper topics the protocol uses:
* static topic for `user-message` message type (also called _contact discovery topic_)
* dynamic topic based on a chat name for `public-group-user-message` message type.
The static topic is always the same and its hex representation is `0xf8946aac`.
In fact, _the contact discovery topic_ is calculated using a dynamic topic
algorithm described below with a constant name `contact-discovery`.
<!-- TODO: Update this, this looks different with partitioned topic -->
Having only one topic for all private chats has an advantage as it's very hard
to figure out who talks to who. A drawback is that everyone receives everyone's
messages but they can decrypt only these they have private keys for.
A dynamic topic is derived from a string using the following algorithm:
Each user publishes periodically to this topic. If user A wants to contact user B, she SHOULD look for their bundle on this contact code topic.
Contact code topic MUST be created following the algorithm below:
```golang
var hash []byte
contactCode := "0x" + hexEncode(activePublicKey) + "-contact-code"
hash = keccak256(name)
var hash []byte = keccak256(name)
var topicLen int = 4
// Whisper V6 specific
var topic [4]byte
topic_len = 4
if len(hash) < topic_len {
topic_len = len(hash)
if len(hash) < topicLen {
topicLen = len(hash)
}
for i = 0; i < topic_len; i++ {
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Partitioned topic
Whisper is broadcast-based protocol. In theory, everyone could communicate using a single topic but that would be extremaly inefficient. Opposite would be using a unique topic for each conversation, however, this brings privacy concerns because it would be much easier to detect whether and when two parties have an active conversation.
Partitioned topics are used to broadcast private messages efficiently. By selecting a number of topic, it is possible to balance efficiency and privacy.
Currently, the number of partitioned topics is set to `5000`. They MUST be generated following the algorithm below:
```golang
var partitionsNum *big.Int = big.NewInt(5000)
var partition *big.Int = big.NewInt(0).Mod(publicKey.X, partitionsNum)
partitionTopic := "contact-discovery-" + strconv.FormatInt(partition.Int64(), 10)
var hash []byte = keccak256(partitionTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
If partitioned topic support is enabled by the Status client, it MUST listen to its paritioned topic. It MUST be generated using the algorithm above and active public key.
### Public chats
A public chat MUST use a topic derived from a public chat name following the algorithm below:
```golang
var hash []byte
hash = keccak256(name)
topicLen = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
### Personal discovery topic
Personal discovery topic is used to ???
A client MUST implement it following the algorithm below:
```golang
personalDiscoveryTopic := "contact-discovery-" + hexEncode(publicKey)
var hash []byte = keccak256(personalDiscoveryTopic)
var topicLen int = 4
if len(hash) < topicLen {
topicLen = len(hash)
}
var topic [4]byte
for i = 0; i < topicLen; i++ {
topic[i] = hash[i]
}
```
Each Status Client SHOULD listen to this topic in order to receive ???
### Generic discovery topic
Generic discovery topic is a legacy topic used to handle all one-to-one chats. The newer implementation should rely on [Partitioned Topic](#partitioned-topic) and [Personal discovery topic](#personal-discovery-topic).
Generic discovery topic MUST be created following [Public chats](#public-chats) topic algorithm using string `contact-discovery` as a name.
### One-to-one topic
In order to receive one-to-one messages incoming from a public key `P`, the Status Client MUST listen to a [Contact Code Topic](#contact-code-topic) created for that public key.
### Group chat topic
Group chats does not have a dedicated topic. All group chat messages (including membership updates) are sent as one-to-one messages to multiple recipients.
## Message encryption
The protocol distinguishes messages encrypted using asymmetric and symmetric encryption.
Even though, the protocol specifies an encryption layer that encrypts messages before passing them to the transport layer, Whisper protocol requires each Whisper message to be encrypted anyway.
Symmetric keys are created using [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) Whisper V6 RPC API method which accepts one param, a string.
Public and group messages are encrypted using symmetric encryption and the key is created from a channel name string. The implementation is available in [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) JSON-RPC method of go-ethereum Whisper implementation.
Messages encrypted with asymmetric encryption should be encrypted using recipient's public key so that only the recipient can decrypt them.
One-to-one messages are encrypted using asymmetric encryption.
Encryption of messages supporting PFS is described in [Perfect Forward Secrecy](#perfect-forward-secrecy-pfs) section.
## Whisper V6 extensions
# Perfect Forward Secrecy (PFS)
### Request historic messages
Additionally to encrypting messages on the Whisper level, the protocol supports PFS specification.
Sends a request for historic messages to a Mailserver. The Mailserver node MUST be a direct peer and MUST be marked a trusted (using `shh_markTrustedPeer`).
A message payload is first encrypted following the PFS specification and then it is encrypted once again following the Whisper specification and this protocol.
The request does not wait for the response. It marely sends a peer-to-peer message to the Mailserver and it's up to Mailserver to process it and start sending historic messages.
As not all messages are encrypted with PFS, a following strategy MAY be used:
1. First, message is decrypted on the Whisper level
2. Try to decrypt the message payload using PFS algorithm
2.1. If successful, pass the decrypted value to (3)
2.2. If failed, pass the unchanged payload to (3)
3. Decode the payload as described in [Payloads](https://github.com/status-im/specs/blob/master/status-payloads-spec.md) specification
The drawback of this approach is that it is impossible to tell which historic messages are the result of which request.
TODO: link to a separate document (currently in the PR).
It's recommended to return messages from newest to oldest. To move further back in time, use `cursor` and `limit`.
[PFS in Status.im docs](https://status.im/research/pfs.html)
#### shhext_requestMessages
# Device syncing
**Parameters**:
1. Object - The message request object:
* `mailServerPeer` - `String`: Mailserver's enode address.
* `from` - `Number` (optional): Lower bound of time range as unix timestamp, default is 24 hours back from now.
* `to` - `Number` (optional): Upper bound of time range as unix timestamp, default is now.
* `limit` - `Number` (optional): Limit the number of messages sent back, default is no limit.
* `cursor` - `String` (optional): Used for paginated requests.
* `topics` - `Array`: hex-encoded message topics.
* `symKeyID` - `String`: an ID of a symmetric key to authenticate to Mailserver, derived from Mailserver password.
TODO: link to a separate document.
# One-to-one messages
One-to-one messages are also known as private messages. These are the messages sent beween two participants of the conversation.
## Sending
Sending a message is fairly easy and relies on the Whisper RPC API, however, some preparation is needed:
1. Obtain a public key of the recipient of the message
2. Add your private key to Whisper using [`shh_addPrivateKey`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_addprivatekey) and save the result as `sigKeyID`
3. Call [`shh_post`(https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_post) with the following settings:
1. `pubKey` MUST be a hex-encoded public key of the message recipient
2. `sig` MUST be set to `sigKeyID`
3. `ttl` MUST be at least `10` (it is in seconds)
4. `topic` MUST be set accordingly to [Topic](#topic) section and hex-encoded
5. `payload` MUST be a hex-encoded string
6. `powTime` MAY be arbitrary but should be enough to perform proof-of-work
7. `powTarget` MUST be equal or higher than `0.002`.
Note: these instructions are for the Whisper V6 RPC API. If you use Whisper service directly or Go `shhclient`, the parameters might have different types.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
### Sending using PFS
When one decides to use PFS, the flow is the same but the payload MUST be additionally encrypted following the [PFS specification](#pfs) before being hex-encoded and passed to `shh_post`.
## Receiving
Receiving private messages depends on Whisper filters idea. Upon receiving, messages are first matched by a topic and then trying to be decrypted using user's private key.
1. Add your private key to Whisper using [`shh_addPrivateKey`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_addprivatekey) and save the result as `sigKeyID`
2. Call [`shh_subscribe`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_subscribe) with criteria:
1. `minPow` MAY be at least `0.002`
2. `topics` MUST be list of hex-encoded topics you expect messages to receive from (follow [Topic](#topic) section)
3. `allowP2P` MUST be set to `true` if offline messages are supported, otherwise can be `false`.
Alternative method is to use [`shh_newMessageFilter`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_newmessagefilter) which takes the same criteria object and then periodically calling [`shh_getFilterMessages`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_getfiltermessages) method.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
# Public messages
Public messages are encrypted with a symmetric key which is publicly known so anyone can participate in the conversation.
The fact that anyone can participate makes the public chats vulnerable to spam attacks. Also, there are no moderators of these chats.
## Sending
1. Calculate a symmetric key using [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) passing a public chat name as a string and save the result to `symKeyID`
2. Call [`shh_post`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_post) with the following settings:
1. `symKeyID` MUST be set to `symKeyID`
2. `sig` MUST be set to `sigKeyID`
3. `ttl` MUST be at least `10` (it is in seconds)
4. `topic` MUST be set accordingly to [Topic](#topic) section and hex-encoded,
5. `payload` MUST be a hex-encoded string,
6. `powTime` MAY be arbitrary but should be enough to perform proof-of-work
7. `powTarget` MUST be equal or higher than `0.002`.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
## Receiving
Receiving public messages depends on Whisper filters idea. Upon receiving, messages are first matched by a topic and then trying to be decrypted using a symmetric key.
1. Calculate a symmetric key using [`shh_generateSymKeyFromPassword`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_generatesymkeyfrompassword) passing public chat name as a string and save the result to `symKeyID`
2. Call [`shh_subscribe`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_subscribe) with criteria:
1. `minPow` MUST be `0.002` at most
2. `topics` MUST be list of hex-encoded topics you expect messages to receive from (follow [Topic](#topic) section)
3. `allowP2P` MUST be set to `true` if offline messages are supported, otherwise can be `false`.
Alternative method is to use [`shh_newMessageFilter`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_newmessagefilter) which takes the same criteria object and then periodically calling [`shh_getFilterMessages`](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API#shh_getfiltermessages) method.
Learn more following [Whisper V6 RPC API](https://github.com/ethereum/go-ethereum/wiki/Whisper-v6-RPC-API).
<!-- TODO: section to describe how to send a group message starting from adding a key in Whisper etc. -->
# Offline messages
A client SHOULD implement mailserver client mode. See [Status Whisper Mailserver Spec](status-whisper-mailserver-spec.md). A Status node MAY implement the server mode as well.
## Whisper V6 extensions (or Status Whisper Node)
Outside of Whisper v6, there are some extensions, message codes and RPC methods that MAY be useful for client implementers. An implementation of this can be found in a fork of Whisper [here](https://github.com/status-im/whisper).
<!--TODO: provide a list of RPC methods from `shhext` API which are relevant to this spec, as well as motivation (rationale section) -->
## Security concerns
TBD.
**Returns**:
`Boolean` - returns `true` if the request was sent.