From 379731cda5cccb741fcbb2293d2ad6baeadf9d0c Mon Sep 17 00:00:00 2001 From: Andrea Maria Piana Date: Mon, 9 Sep 2019 10:13:29 +0200 Subject: [PATCH 1/3] Add payload specs --- status-group-chats-spec.md | 1 + status-payloads-spec.md | 164 ++++++++++++++++++++++++++----------- 2 files changed, 119 insertions(+), 46 deletions(-) create mode 100644 status-group-chats-spec.md diff --git a/status-group-chats-spec.md b/status-group-chats-spec.md new file mode 100644 index 0000000..463d4a3 --- /dev/null +++ b/status-group-chats-spec.md @@ -0,0 +1 @@ +# Group chat specs diff --git a/status-payloads-spec.md b/status-payloads-spec.md index cfc641c..13a3fc6 100644 --- a/status-payloads-spec.md +++ b/status-payloads-spec.md @@ -21,12 +21,23 @@ as various clients created using different technologies. - [Introduction](#introduction) - [Payload wrapper](#payload-wrapper) - [Encoding](#encoding) - - [Message](#message) - - [Payload](#payload) - - [Content types](#content-types) - - [Message types](#message-types) - - [Clock vs Timestamp and message ordering](#clock-vs-timestamp-and-message-ordering) - - [Chats](#chats) + - [Types of Messages] (#types-of-messages) + - [Message](#message) + - [Payload](#payload) + - [Content types](#content-types) + - [Message types](#message-types) + - [Clock vs Timestamp and message ordering](#clock-vs-timestamp-and-message-ordering) + - [Chats](#chats) + - [Contact requests](#contact-requests) + - [Payload] (#payload) + - [Contact update] (#contact-update) + - [Handling contact messages] (#handling-contact-messages) + - [SyncInstallation](#sync-installation) + - [Payload](#payload) + - [PairInstallation](#pair-installation) + - [Payload](#payload) + - [GroupMembershipUpdate](#group-membership-update) + - [Payload](#payload) - [Upgradability](#upgradability) - [Security Considerations](#security-considerations) - [Design rationale](#design-rationale) @@ -56,43 +67,28 @@ If a signature is not present but an author is provided by a layer below, the me The payload is encoded using [Transit format](https://github.com/cognitect/transit-format). Transit was chosen over JSON in order to reduce the bandwidth. -Example of a valid encoded payload: +## Types of messages -``` -["~#c4",["abc123","text/plain","~:public-group-user-message",154593077368201,1545930773682,["^ ","~:chat-id","testing-adamb","~:text","abc123"]]] -``` +### Message -The message is an array and each index value has its meaning: -* 0: `c4` is a decoder handler identification for the current payload format. Identifications allow to register handlers for many different types of payload -* 1: array which items correspond to the described payload fields above +The type `Message` represents a text message exchanged between clients and is identified by the transit tag `c4`. -For more details regarding serialization and deserialization please consult [transit format](https://github.com/cognitect/transit-format) specification. - - - -## Message - -The type `Message` represents a text message exchanged between clients. - - - -### Payload +#### Payload Payload is a struct (a compound data type) with the following fields (order is important): -| Field | Name | Type | -| ----- | ---- | ---- | -| 1 | text | `string` | -| 2 | content type | `enum` (more in [Content types](#content-types)) | -| 3 | message type | `enum` (more in [Message types](#message-types)) | -| 4 | clock | `int64` | -| 5 | timestamp | `int64` | -| 6 | content | `struct { chat-id string, text string }` | +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | text | `string` | The text version of the message content | +| 2 | content type | `enum` (more in [Content types](#content-types)) | See details | +| 3 | message type | `enum` (more in [Message types](#message-types)) | See details | +| 4 | clock | `int64` | See details | +| 5 | timestamp | `int64` | See details | +| 6 | content | `struct { chat-id string, text string, response-to string }` | The chat-id of the chat this message is destined to, the text of the content and optionally the id of the message it is responding to| -### Content types +#### Content types Content types are required for a proper interpretation of incoming messages. Not each message is a plain text but may carry a different information. @@ -110,7 +106,7 @@ These are currently underspecified. We refer to real-world implementations for c -### Message types +#### Message types Message types are required to decide how a particular message is encrypted and what metadata needs to be attached when passing a message to the transport layer. For more on this, see [Status Whisper Usage Specification](./status-whisper-usage-spec.md). @@ -122,19 +118,15 @@ The following messages types MUST be supported: * `user-message` is a private message * `group-user-message` is a message to the private group. -### Clock vs Timestamp and message ordering +#### Clock vs Timestamp and message ordering -`timestamp` MUST be Unix time calculated when the message is created. Because the peers in the Whisper network should have synchronized time, `timestamp` values should be fairly accurate among all Whisper network participants. +`timestamp` MUST be Unix time calculated when the message is created in milliseconds. This field SHOULD not be relied upon for message ordering. `clock` SHOULD be calculated using the algorithm of [Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps). When there are messages available in a chat, `clock`'s value is calculated based on the last received message in a particular chat: `last-message-clock-value + 1`. If there are no messages, `clock` is initialized with `timestamp * 100`'s value. `clock` value is used for the message ordering. Due to the used algorithm and distributed nature of the system, we achieve casual ordering which might produce counterintuitive results in some edge cases. For example, when one joins a public chat and sends a message before receiving the exist messages, their message `clock` value might be lower and the message will end up in the past when the historical messages are fetched. - - -## Chats - +#### Chats Chat is a structure that helps organize messages. It's usually desired to display messages only from a single recipient or a group of recipients at a time and chats help to achieve that. @@ -149,10 +141,91 @@ All incoming messages can be matched against a chat. Below you can find a table +### Contact Requests + +These messages are used to notify the receiving end that it has been added to the sender's contact. They are identified by the transit tags `c2`, `c3`, `c4` respectively, but they are all interchangeable, meaning a client SHOULD handle them in exactly the same way. + +#### Payload + +Payload is a struct (a compound data type) with the following fields (order is important): + + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | name | `string` | The self-assigned name of the user (DEPRECATED) | +| 2 | profile image | `string` | The base64 encoded profile picture of the user | +| 3 | address | `string` | The ethereum address of the user | +| 4 | fcm-token | `string` | The FCM Token used by mobile devices for push notifications (DEPRECATED) | +| 5 | device-info | `[struct { id string, fcm-token string }]` | A list of pair `installation-id`, `fcm-token` for each device that is currently paired | + +#### Contact update + +A client SHOULD send a `ContactUpdate` to all the contacts each time: + +- The name is edited +- The profile image is edited +- A new device has been paired + +A client SHOULD also periodically send a `ContactUpdate` to all the contacts, the interval is up to the client. + + +#### Handling contact messages + +A client SHOULD handle any `Contact*` message in the same way. Any `Contact*` message with a whisper timestamp lower than the last one processed MUST be discarded. + +### SyncInstallation + +`SyncInstallation` messages are used to synchronize in a best-effort way all the paired installations. It is identified by a transit tag of `p1` + +#### Payload + +Payload is a struct (a compound data type) with the following fields (order is important): + + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1| contacts | `[struct { name string last-updated int device-info struct {id string fcm-token string } pending? bool}` | An array of contacts | +| 2 | account | `struct {name string photo-path string last-updated int}` | Information about your own account | +| 3 | chat | `struct {:public? bool :chat-id string}` | A description of a public chat opened by the client | + +### PairInstallation + +`PairInstallation` messages are used to propagate informations about a device to its paired devices. It is identified by a transit tag of `p2` + +#### Payload + +Payload is a struct (a compound data type) with the following fields (order is important): + + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1| installation-id | `string` | A randomly generated id that identifies this device | +| 2 | device-type | `string` | The OS of the device `ios`,`android` or `desktop` | +| 3 | name | `string` | The self-assigned name of the device | +| 4 | fcm-token | `string` | The FCM Token used by mobile devices for push notifications | + +### GroupMembershipUpdate + +`GroupMembershipUpdate` is a message used to propagate information about group membership changes in a group chat.. It is identified by a transit tag of `g5`. +The details are in the [Group chats specs](status-group-chats-spec.md) + +#### Payload + +Payload is a struct (a compound data type) with the following fields (order is important): + + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1| chat-id | `string` | The chat id of the chat where the change is to take place | +| 2 | membership-updates | See details | A list of events that describe the membership changes | +| 3 | message | `Transit message` | An optional message, described in [Message](#message) | + ## Upgradability -The current protocol format is hardly upgradable without breaking backward compatibility. Because Transit is used in this particular way described above, the only reliable option is to append a new field to the Transit record definition. It will be simply ignored by the old clients. - +There are two ways to upgrade the protocol without breaking compatibility: + +- Map fields can be enriched with a new key, which will be ignored by old clients. +- An element can be appended to the `Transit` array, which will also be ignored by old clients. ## Security Considerations @@ -162,5 +235,4 @@ TBD. ### Why are you using Transit and Protobuf? -Transit was initially chose for encoding, and Protobuf was added afterwards. This is partly due to the history of the protocol living inside of `status-react`, which is written in Clojurescript. In future versions of payload and data sync client specifications it is likely we'll move towards Protobuf only. See e.g. [Dasy](https://github.com/vacp2p/dasy) for a research proof of concept. - +Transit was initially chose for encoding, and Protobuf was added afterwards. This is partly due to the history of the protocol living inside of `status-react`, which is written in Clojurescript. From 90e0428703bd8958fe12bfe833629252c527fd9a Mon Sep 17 00:00:00 2001 From: Andrea Maria Piana Date: Mon, 9 Sep 2019 13:36:31 +0200 Subject: [PATCH 2/3] add group chat specs --- status-group-chats-spec.md | 203 ++++++++++++++++++++++++++++++++++++- 1 file changed, 202 insertions(+), 1 deletion(-) diff --git a/status-group-chats-spec.md b/status-group-chats-spec.md index 463d4a3..b8694aa 100644 --- a/status-group-chats-spec.md +++ b/status-group-chats-spec.md @@ -1 +1,202 @@ -# Group chat specs +# Status Group Chat Specification + +> Version: 0.1 (Draft) +> +> Authors: Andrea Maria Piana +> + + +## Table of Contents + +- [Abstract](#abstract) +- [Membership updates](#membership-updates) + - [Chat ID](#chat-id) + - [Signature](#signature) + - [Group membership event](#group-membership-event) + - [chat-created](#chat-created) + - [name-changed](#name-changed) + - [members-added](#members-added) + - [members-joined](#members-joined) + - [admins-added](#admins-added) + - [members-removed](#members-removed) + - [admin-removed](#admin-removed) + + +## Abstract + +This documents describes the group chat protocol used by the status application. Pairwise encryption is used among member so a message is exchanged between each participants, similarly to a one-to-one message. + +## Membership updates + +Membership updates messages are used to propagate group chat membership changes. The transit format is described in the [Status Payload Specs](status-payload-specs.md). Here we will be describing each specific field. + +The format is: + +``` +{ + "events": [], + "signature": string, + "chat-id": string +} +``` + +### Chat ID + +Each membership update MUST be sent with a corresponding `chat-id`. +The format of this chat id MUST be a string, [UUID](https://tools.ietf.org/html/rfc4122 ), concatenated with the hex-encoded public key of the creator of the chat. This chat-id MUST be validated by all clients, and should be discarded if it does not follow these rules. + +### Signature + +The signature for each event is calculated by creating a JSON array of all the `events` sorted by `clock-value` in ascending order, where each event is transformed in an array of tuples `field-name`, `value`. The last element of the array MUST be the `chat-id`. + +For example the event: + +``` + { + "chat-id": "chat-id", + "events": [ + {"b": "b-value" + "clock-value": 1, + "a": "a-value" + }, + { + "e": "e-value", + "clock-value": 0, + "a": "a-value" + } + ] + } + +``` + +Results in the structure: + +``` + [ + [ + [ + ["a" "a-value"], + ["clock-value", 0], + ["e" "e-value"] + ], + [ + ["a", "a-value"], + ["b", "b-value"], + ["clock-value", 1] + ] + ], + "chat-id" + ] +``` + +This structure is then stringified and the `Keccak256` of the string is then signed using its private key by the author and added to the payload. + + +### Group membership event + +Any group membership event receive MUST be verified using by calculating the signature as in the method described above, and the author MUST be extracted from it, if the verification fails the event MUST be discarded. + +#### chat-created + +``` +{ + "type": "chat-created", + "name": string + "clock-value": uint +} +``` + + +Chat created event is the first event that needs to be sent. Any event with a clock value lower then this MUST be discarded. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates and create a chat with identified by `chat-id` and named `name`. + +#### name-changed + +``` +{ + "type": "name-changed" + "name": string + "clock-value": uint +} +``` + +A name changed event is used by admins to change the name of the group chat. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored. +If the event is valid the chat name SHOULD be changed to `name`. + + +#### members-added + +``` +{ + "type": "members-added" + "members": [string] + "clock-value": uint +} +``` + +A members added event is used by admins to add members to the chat. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored. +If the event is valid a client SHOULD update the list of members of the chat who have not joined, adding the `members` received. +`members` is an array of hex encoded public keys. + +#### member-joined + +``` +{ + "type": "member-joined" + "member": string + "clock-value": uint +} +``` + +A members joined event is used by a member of the chat to signal that they want to start receiving messages from this chat. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates and MUST ensure the author of the event is the same as the one specified by the `member` field. +If the event is valid a client SHOULD update the list of members of the chat who joined, adding `member`. Any `message` sent to the group chat should now include the newly joined member. + +#### admins-added + +``` +{ + "type": "admins-added" + "members": [string] + "clock-value": uint +} +``` + +An admins added event is used by admins to add make other admins in the chat. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates, MUST ensure the author of the event is an admin of the chat and MUST ensure all `members` are already `members` of the chat, otherwise the event MUST be ignored. +If the event is valid a client SHOULD update the list of admins of the chat, adding the `members` received. +`members` is an array of hex encoded public keys. + +#### member-removed + +``` +{ + "type": "member-removed" + "member": string + "clock-value": uint +} +``` + +A member-removed event is used to leave or kick members of the chat. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates, MUST ensure that: +- If the author of the event is an admin, target can only be themselves or a non-admin member. +- If the author of the event is not an admin, the target of the event can only be themselves. +- +If the event is valid a client SHOULD remove the member from the list of `members`/`admins` of the chat, and no further message should be sent to them. + +#### admin-removed + +``` +{ + "type": "admin-removed" + "member": string + "clock-value": uint +} +``` + +An admin-removed event is used to drop admin privileges. +Upon receiving this event a client MUST validate the `chat-id` provided with the updates, MUST ensure that the author of the event is also the target of the event. + +If the event is valid a client SHOULD remove the member from the list of `admins` of the chat. From 51300922ac0b8652922e00c4aceecb103cefc31b Mon Sep 17 00:00:00 2001 From: Andrea Maria Piana Date: Tue, 10 Sep 2019 09:06:57 +0200 Subject: [PATCH 3/3] address feedback --- status-group-chats-spec.md | 22 ++++++++++++---------- status-payloads-spec.md | 23 ++++++----------------- 2 files changed, 18 insertions(+), 27 deletions(-) diff --git a/status-group-chats-spec.md b/status-group-chats-spec.md index b8694aa..d885584 100644 --- a/status-group-chats-spec.md +++ b/status-group-chats-spec.md @@ -34,7 +34,7 @@ The format is: ``` { - "events": [], + "events": [struct {"type": string, "member": string, "members": [string], "clock-value": uint, "name": string], "signature": string, "chat-id": string } @@ -43,11 +43,12 @@ The format is: ### Chat ID Each membership update MUST be sent with a corresponding `chat-id`. -The format of this chat id MUST be a string, [UUID](https://tools.ietf.org/html/rfc4122 ), concatenated with the hex-encoded public key of the creator of the chat. This chat-id MUST be validated by all clients, and should be discarded if it does not follow these rules. +The format of this chat id MUST be a string, [UUID](https://tools.ietf.org/html/rfc4122 ), concatenated with the hex-encoded public key of the creator of the chat. This chat-id MUST be validated by all clients, and MUST be discarded if it does not follow these rules. ### Signature -The signature for each event is calculated by creating a JSON array of all the `events` sorted by `clock-value` in ascending order, where each event is transformed in an array of tuples `field-name`, `value`. The last element of the array MUST be the `chat-id`. +The signature for each event is calculated by creating a JSON array of all the `events` sorted by `clock-value` in ascending order, where each event is transformed in an array of tuples `field-name`, `value`, sorted by `field-name` in ascending alphabetical order. The last element of the array MUST be the `chat-id`. +Empty fields MUST be removed. For example the event: @@ -89,12 +90,13 @@ Results in the structure: ] ``` -This structure is then stringified and the `Keccak256` of the string is then signed using its private key by the author and added to the payload. +This structure is then stringified collapsing all whitespaces and the `Keccak256` of the string is then signed using its private key by the author and added to the payload. ### Group membership event -Any group membership event receive MUST be verified using by calculating the signature as in the method described above, and the author MUST be extracted from it, if the verification fails the event MUST be discarded. +Any group membership event received MUST be verified by calculating the signature as per the method described above. +The author MUST be extracted from it, if the verification fails the event MUST be discarded. #### chat-created @@ -137,7 +139,7 @@ If the event is valid the chat name SHOULD be changed to `name`. A members added event is used by admins to add members to the chat. Upon receiving this event a client MUST validate the `chat-id` provided with the updates and MUST ensure the author of the event is an admin of the chat, otherwise the event MUST be ignored. -If the event is valid a client SHOULD update the list of members of the chat who have not joined, adding the `members` received. +If the event is valid a client MUST update the list of members of the chat who have not joined, adding the `members` received. `members` is an array of hex encoded public keys. #### member-joined @@ -152,7 +154,7 @@ If the event is valid a client SHOULD update the list of members of the chat who A members joined event is used by a member of the chat to signal that they want to start receiving messages from this chat. Upon receiving this event a client MUST validate the `chat-id` provided with the updates and MUST ensure the author of the event is the same as the one specified by the `member` field. -If the event is valid a client SHOULD update the list of members of the chat who joined, adding `member`. Any `message` sent to the group chat should now include the newly joined member. +If the event is valid a client MUST update the list of members of the chat who joined, adding `member`. Any `message` sent to the group chat should now include the newly joined member. #### admins-added @@ -166,7 +168,7 @@ If the event is valid a client SHOULD update the list of members of the chat who An admins added event is used by admins to add make other admins in the chat. Upon receiving this event a client MUST validate the `chat-id` provided with the updates, MUST ensure the author of the event is an admin of the chat and MUST ensure all `members` are already `members` of the chat, otherwise the event MUST be ignored. -If the event is valid a client SHOULD update the list of admins of the chat, adding the `members` received. +If the event is valid a client MUST update the list of admins of the chat, adding the `members` received. `members` is an array of hex encoded public keys. #### member-removed @@ -184,7 +186,7 @@ Upon receiving this event a client MUST validate the `chat-id` provided with the - If the author of the event is an admin, target can only be themselves or a non-admin member. - If the author of the event is not an admin, the target of the event can only be themselves. - -If the event is valid a client SHOULD remove the member from the list of `members`/`admins` of the chat, and no further message should be sent to them. +If the event is valid a client MUST remove the member from the list of `members`/`admins` of the chat, and no further message should be sent to them. #### admin-removed @@ -199,4 +201,4 @@ If the event is valid a client SHOULD remove the member from the list of `member An admin-removed event is used to drop admin privileges. Upon receiving this event a client MUST validate the `chat-id` provided with the updates, MUST ensure that the author of the event is also the target of the event. -If the event is valid a client SHOULD remove the member from the list of `admins` of the chat. +If the event is valid a client MUST remove the member from the list of `admins` of the chat. diff --git a/status-payloads-spec.md b/status-payloads-spec.md index 13a3fc6..81f8ca8 100644 --- a/status-payloads-spec.md +++ b/status-payloads-spec.md @@ -90,7 +90,7 @@ Payload is a struct (a compound data type) with the following fields (order is i #### Content types -Content types are required for a proper interpretation of incoming messages. Not each message is a plain text but may carry a different information. +Content types are required for a proper interpretation of incoming messages. Not each message is plain text but may carry a different information. The following content types MUST be supported: * `text/plain` identifies a message which content is a plain text. @@ -143,18 +143,16 @@ All incoming messages can be matched against a chat. Below you can find a table ### Contact Requests -These messages are used to notify the receiving end that it has been added to the sender's contact. They are identified by the transit tags `c2`, `c3`, `c4` respectively, but they are all interchangeable, meaning a client SHOULD handle them in exactly the same way. +Contact requests consists in 3 kind of messages: `ContactRequest`, `ContactRequestConfirmed` and `ContactUpdate`. +These messages are used to notify the receiving end that it has been added to the sender's contact. They are identified by the transit tags `c2`, `c3`, `c4` respectively, but they are all interchangeable, meaning a client SHOULD handle them in exactly the same way. The payload of the 3 messages is identical. #### Payload -Payload is a struct (a compound data type) with the following fields (order is important): - - | Field | Name | Type | Description | | ----- | ---- | ---- | ---- | | 1 | name | `string` | The self-assigned name of the user (DEPRECATED) | | 2 | profile image | `string` | The base64 encoded profile picture of the user | -| 3 | address | `string` | The ethereum address of the user | +| 3 | address | `string` | The ethereum address of the user | | 4 | fcm-token | `string` | The FCM Token used by mobile devices for push notifications (DEPRECATED) | | 5 | device-info | `[struct { id string, fcm-token string }]` | A list of pair `installation-id`, `fcm-token` for each device that is currently paired | @@ -166,7 +164,7 @@ A client SHOULD send a `ContactUpdate` to all the contacts each time: - The profile image is edited - A new device has been paired -A client SHOULD also periodically send a `ContactUpdate` to all the contacts, the interval is up to the client. +A client SHOULD also periodically send a `ContactUpdate` to all the contacts, the interval is up to the client, the Status official client sends these updates every 48 hours. #### Handling contact messages @@ -179,9 +177,6 @@ A client SHOULD handle any `Contact*` message in the same way. Any `Contact*` me #### Payload -Payload is a struct (a compound data type) with the following fields (order is important): - - | Field | Name | Type | Description | | ----- | ---- | ---- | ---- | | 1| contacts | `[struct { name string last-updated int device-info struct {id string fcm-token string } pending? bool}` | An array of contacts | @@ -194,9 +189,6 @@ Payload is a struct (a compound data type) with the following fields (order is i #### Payload -Payload is a struct (a compound data type) with the following fields (order is important): - - | Field | Name | Type | Description | | ----- | ---- | ---- | ---- | | 1| installation-id | `string` | A randomly generated id that identifies this device | @@ -211,9 +203,6 @@ The details are in the [Group chats specs](status-group-chats-spec.md) #### Payload -Payload is a struct (a compound data type) with the following fields (order is important): - - | Field | Name | Type | Description | | ----- | ---- | ---- | ---- | | 1| chat-id | `string` | The chat id of the chat where the change is to take place | @@ -224,7 +213,7 @@ Payload is a struct (a compound data type) with the following fields (order is i There are two ways to upgrade the protocol without breaking compatibility: -- Map fields can be enriched with a new key, which will be ignored by old clients. +- Struct fields can be enriched with a new key, which will be ignored by old clients. - An element can be appended to the `Transit` array, which will also be ignored by old clients. ## Security Considerations