spec/mail-server (#77)

* mail server specification

* todo

* Update waku.md

* add adam

* changes

* remove todo

* reference

* fixes

* updates

README.md

@@ -14,6 +14,7 @@ These protocols define various components of the [vac](https://vac.dev) stack.
  - [remote log](./remote-log.md) - Remote replication of local logs.
  - [mdf](./mdf.md) - Metadata field for [MVDS](./mvds.md) messages. 
  - [waku](./waku.md) - ÐΞVp2p wire protocol, substituting [EIP-627](https://eips.ethereum.org/EIPS/eip-627).
+ - [wms](./wms.md) - Mailserver specification for archiving and delivering historical [waku](./waku.md) envelopes on demand.
 
 
 ## Style guide

waku.md

@@ -9,7 +9,6 @@
 - [Abstract](#abstract)
 - [Motivation](#motivation)
 - [Definitions](#definitions)
-- [Roles](#roles)
 - [Specification](#specification)
     - [Use of RLPx transport protocol](#use-of-rlpx-transport-protocol)
     - [ABNF specification](#abnf-specification)
@@ -20,9 +19,6 @@
     - [Packet code Rationale](#packet-code-rationale)
 - [Additional capabilities](#additional-capabilities)
     - [Light node](#light-node)
-    - [Mailserver and client](#mailserver-and-client)
-        - [Requesting messages](#requesting-messages)
-        - [Receiving historic messages](#receiving-historic-messages)
     - [Accounting for resources](#accounting-for-resources)
 - [Backwards Compatibility](#backwards-compatibility)
     - [Waku-Whisper bridging](#waku-whisper-bridging)
@@ -57,13 +53,6 @@ Waku was created to incrementally improve in areas that Whisper is lacking in, w
 | **Envelope**    | Messages sent and received by Waku nodes.              |
 | **Node**        | Some process that is able to communicate for Waku.    |
 
-## Roles
-
-| Role            | Definition                                       |
-| --------------- | ------------------------------------------------ |
-| **Mail Server** | A node responsible for archiving messages.       |
-| **Mail Client** | A node that requests messages from mail servers. |
-
 ## Underlying Transports and Prerequisites
 
 ### Use of DevP2P
@@ -72,7 +61,7 @@ For nodes to communicate, they MUST implement devp2p and run RLPx. They MUST hav
 
 ### Gossip based routing
 
-In Whisper, messages are gossiped between peers. Whisper is a form of rumor-mongering protocol that works by flooding to its connected peers based on some factors. Messages are elgible for retransmission until their TTL expires. A node SHOULD relay messages to all connected nodes if an envelope matches their PoW and bloom filter settings. If a node works in light mode, it MAY choose not to forward envelopes. A node MUST NOT send expired envelopes, unless the envelopes are sent as a mailserver response. A node SHOULD NOT send a message to a peer that it has already sent before.
+In Whisper, messages are gossiped between peers. Whisper is a form of rumor-mongering protocol that works by flooding to its connected peers based on some factors. Messages are elgible for retransmission until their TTL expires. A node SHOULD relay messages to all connected nodes if an envelope matches their PoW and bloom filter settings. If a node works in light mode, it MAY choose not to forward envelopes. A node MUST NOT send expired envelopes, unless the envelopes are sent as a [mailserver](./wms.md) response. A node SHOULD NOT send a message to a peer that it has already sent before.
 
 ## Wire Specification
 
@@ -338,7 +327,9 @@ Packet codes `0x7E` and `0x7F` may be used to implement Waku Mail Server and Cli
 
 ## Additional capabilities
 
-Waku supports multiple capabilities. These include light node, rate limiting, mailserver (client and server) and bridging of traffic. Here we list these capabilities, how they are identified, what properties they have and what invariants they must maintain.
+Waku supports multiple capabilities. These include light node, rate limiting and bridging of traffic. Here we list these capabilities, how they are identified, what properties they have and what invariants they must maintain.
+
+Additionally there is the capability of a mailserver which is documented in its on [specification](./wms). 
 
 ### Light node
 
@@ -348,40 +339,6 @@ Light nodes MUST NOT forward any incoming messages, they MUST only send their ow
 
 Light nodes are identified by the `light_node` value in the status message.
 
-### Mailserver and client
-
-Mailservers are waku nodes that can archive messages and deliver them to its peers on-demand. A node which wants to provide mailserver functionality MUST store envelopes from incoming message packets (Waku packet-code 0x01). The envelopes can be stored in any format, however they MUST be serialized and deserialized to the Waku envelope format.
-
-A mailserver SHOULD store envelopes for all topics to be generally useful for any peer, however for specific use cases it MAY store envelopes for a subset of topics.
-
-#### Requesting messages
-
-In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing mailserver functionality. This packet requires one argument which MUST be a Waku envelope.
-
-In the Waku envelope's payload section, there MUST be RLP-encoded information about the details of the request:
-
-```
-[ Lower, Upper, Bloom, Limit, Cursor ]
-```
-
-`Lower`: 4-byte wide unsigned integer (UNIX time in seconds; oldest requested envelope's creation time)
-`Upper`: 4-byte wide unsigned integer (UNIX time in seconds; newest requested envelope's creation time)
-`Bloom`: 64-byte wide array of Waku topics encoded in a bloom filter to filter envelopes
-`Limit`: 4-byte wide unsigned integer limiting the number of returned envelopes
-`Cursor`: 32-byte wide array of a cursor returned from the previous request (optional)
-
-The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit` so that the requester can send another request using the obtained `Cursor` value. What exactly is in the `Cursor` is up to the implementation. The requester SHOULD NOT use a `Cursor` obtained from one mailserver in a request to another mailserver because the format or the result MAY be different.
-
-The envelope MUST be signed with a symmetric key agreed between the requester and Mailserver.
-
-#### Receiving historic messages
-
-Historic messages MUST be sent to a peer as a packet with a P2P Message code (`0x7f`) followed by an array of Waku envelopes.
-
-In order to receive historic messages from a mailserver, a node MUST trust the selected mailserver, that is allow to receive expired packets with the P2P Message code. By default, such packets are discarded.
-
-Received envelopes MUST be passed through the Waku envelopes pipelines so that they are picked up by registered filters and passed to subscribers.
-
 ### Accounting for resources (experimental)
 
 Nodes MAY implement accounting, keeping track of resource usage. It is heavily inspired by Swarm's [SWAP protocol](https://www.bokconsulting.com.au/wp-content/uploads/2016/09/tron-fischer-sw3.pdf), and works by doing pairwise accounting for resources.
@@ -423,7 +380,7 @@ It is desirable to have a strategy for maintaining forward compatibility between
 
 ## Appendix A: Security considerations
 
-There are several security considerations to take into account when running Waku. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used, such as mailserver, light node, and so on.
+There are several security considerations to take into account when running Waku. Chief among them are: scalability, DDoS-resistance and privacy. These also vary depending on what capabilities are used. The security considerations for extra capabilities such as [mailservers](./wms.md#security-considerations) can be found in their respective specifications.
 
 ### Scalability and UX
 
@@ -431,10 +388,6 @@ There are several security considerations to take into account when running Waku
 
 In version 0 of Waku, bandwidth usage is likely to be an issue. For more investigation into this, see the theoretical scaling model described [here](https://github.com/vacp2p/research/tree/dcc71f4779be832d3b5ece9c4e11f1f7ec24aac2/whisper_scalability).
 
-**Mailserver High Availability requirement:**
-
-A mailserver has to be online to receive messages for other nodes, this puts a high availability requirement on it.
-
 **Gossip-based routing:**
 
 Use of gossip-based routing doesn't necessarily scale. It means each node can see a message multiple times, and having too many light nodes can cause propagation probability that is too low. See [Whisper vs PSS](https://our.status.im/whisper-pss-comparison/) for more and a possible Kademlia based alternative.
@@ -453,10 +406,6 @@ The main privacy concern with light nodes is that directly connected peers will
 
 By having a bloom filter where only the topics you are interested in are set, you reveal which messages you are interested in. This is a fundamental tradeoff between bandwidth usage and privacy, though the tradeoff space is likely suboptimal in terms of the [Anonymity](https://eprint.iacr.org/2017/954.pdf) [trilemma](https://petsymposium.org/2019/files/hotpets/slides/coordination-helps-anonymity-slides.pdf).
 
-**Mailserver client privacy:**
-
-A mailserver client has to trust a mailserver, which means they can send direct traffic. This reveals what topics / bloom filter a node is interested in, along with its peerID (with IP).
-
 **Privacy guarantees not rigorous:**
 
 Privacy for Whisper / Waku haven't been studied rigorously for various threat models like global passive adversary, local active attacker, etc. This is unlike e.g. Tor and mixnets.
@@ -471,10 +420,6 @@ Similar to bloom filter privacy, if you use a very specific topic you reveal mor
 
 Proof of work is a poor spam prevention mechanism. A mobile device can only have a very low PoW in order not to use too much CPU / burn up its phone battery. This means someone can spin up a powerful node and overwhelm the network.
 
-**Mailserver trusted connection:**
-
-A mailserver has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning mailserver can overwhelm an individual node.
-
 ### Censorship resistance
 
 **Devp2p TCP port blockable:**

wms.md

@@ -0,0 +1,88 @@
+# Waku Mailserver
+
+> Version 0.1.0
+>
+> Authors: Adam Babik <adam@status.im>, Dean Eigenmann <dean@status.im>, Oskar Thorén <oskar@status.im> (alphabetical order)
+
+## Table of Contents
+
+1. [Abstract](#abstract)
+2. [Specification](#specification)
+    1. [Requesting messages](#requesting-messages)
+    2. [Receiving historic messages](#receiving-historic-messages)
+    3. [Security considerations](#security-considerations)
+3. [Copyright](#copyright)
+
+## Abstract
+
+In this specification, we describe Mailservers. These are nodes responsible for archiving messages and delivering them too peers on-demand.
+
+## Specification
+
+A node which wants to provide mailserver functionality MUST store envelopes from incoming message packets (Waku packet-code 0x01). The envelopes can be stored in any format, however they MUST be serialized and deserialized to the Waku envelope format.
+
+A mailserver SHOULD store envelopes for all topics to be generally useful for any peer, however for specific use cases it MAY store envelopes for a subset of topics.
+
+### Requesting messages
+
+In order to request historic messages, a node MUST send a packet P2P Request (`0x7e`) to a peer providing mailserver functionality. This packet requires one argument which MUST be a Waku envelope.
+
+In the Waku envelope's payload section, there MUST be RLP-encoded information about the details of the request:
+
+```
+[ Lower, Upper, Bloom, Limit, Cursor ]
+```
+
+| Field    |   Description                                                                                    |
+| -------- | ------------------------------------------------------------------------------------------------ |
+| `Lower`  |   4-byte wide unsigned integer (UNIX time in seconds; oldest requested envelope's creation time) |
+| `Upper`  |   4-byte wide unsigned integer (UNIX time in seconds; newest requested envelope's creation time) |
+| `Bloom`  |   64-byte wide array of Waku topics encoded in a bloom filter to filter envelopes                |
+| `Limit`  |   4-byte wide unsigned integer limiting the number of returned envelopes                         |
+| `Cursor` |   32-byte wide array of a cursor returned from the previous request (optional)                   |
+
+The `Cursor` field SHOULD be filled in if a number of envelopes between `Lower` and `Upper` is greater than `Limit` so that the requester can send another request using the obtained `Cursor` value. What exactly is in the `Cursor` is up to the implementation. The requester SHOULD NOT use a `Cursor` obtained from one mailserver in a request to another mailserver because the format or the result MAY be different.
+
+The envelope MUST be encrypted with a symmetric key agreed between the requester and Mailserver.
+
+### Receiving historic messages
+
+Historic messages MUST be sent to a peer as a packet with a P2P Message code (`0x7f`) followed by an array of Waku envelopes.
+
+In order to receive historic messages from a mailserver, a node MUST trust the selected mailserver, that is allow to receive expired packets with the P2P Message code. By default, such packets are discarded.
+
+Received envelopes MUST be passed through the Whisper envelope pipelines so that they are picked up by registered filters and passed to subscribers.
+
+For a requester, to know that all messages have been sent by mailserver, it SHOULD handle P2P Request Complete code (`0x7d`). This code is followed by a list with:
+
+```
+[ RequestID, LastEnvelopeHash, Cursor ]
+```
+
+| Field              | Description                                                                                |
+| ------------------ | ------------------------------------------------------------------------------------------ |
+| `RequestID`        | 32-byte wide array with a Keccak-256 hash of the envelope containing the original request. |
+| `LastEnvelopeHash` | 32-byte wide array with a Keccak-256 hash of the last sent envelope for the request.       |
+| `Cursor`           | an array of a cursor returned from the previous request (optional)                         |
+
+If `Cursor` is not empty, it means that not all messages were sent due to the set `Limit` in the request. One or more consecutive requests MAY be sent with `Cursor` field filled in in order to receive the rest of the messages.
+
+### Security considerations
+
+There are several security considerations to take into account when running or interacting with Mailservers. Chief among them are: scalability, DDoS-resistance and privacy.
+
+**Mailserver High Availability requirement:**
+
+A mailserver has to be online to receive messages for other nodes, this puts a high availability requirement on it.
+
+**Mailserver client privacy:**
+
+A mailserver client has to trust a mailserver, which means they can send direct traffic. This reveals what topics / bloom filter a node is interested in, along with its peerID (with IP).
+
+**Mailserver trusted connection:**
+
+A mailserver has a direct TCP connection, which means they are trusted to send traffic. This means a malicious or malfunctioning mailserver can overwhelm an individual node.
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).