From 1b909c3d9f4323b71f78c7c65dc20669b1ef4534 Mon Sep 17 00:00:00 2001 From: Andrea Maria Piana Date: Thu, 30 Apr 2020 12:24:58 +0200 Subject: [PATCH 1/3] Add draft payloads --- docs/draft/6-payloads.md | 335 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 335 insertions(+) create mode 100644 docs/draft/6-payloads.md diff --git a/docs/draft/6-payloads.md b/docs/draft/6-payloads.md new file mode 100644 index 0000000..20331d9 --- /dev/null +++ b/docs/draft/6-payloads.md @@ -0,0 +1,335 @@ +--- +permalink: /spec/6 +parent: Stable specs +title: 6/PAYLOADS +--- + +# 6/PAYLOADS + +> Version: 0.2 +> +> Status: Stable +> +> Authors: Adam Babik , Andrea Maria Piana , Oskar Thorén (alphabetical order) + +## Abstract + +This specifications decribes how the payload of each message in Status looks +like. It is primarly centered around chat and chat-related use cases. + +The payloads aims be flexible enough to support messaging but also cases +described in the [Status Whitepaper](https://status.im/whitepaper.pdf) as well +as various clients created using different technologies. + +## Table of Contents + +- [Status Message Payloads Specification](#status-message-payloads-specification) + - [Abstract](#abstract) + - [Table of Contents](#table-of-contents) + - [Introduction](#introduction) + - [Payload wrapper](#payload-wrapper) + - [Encoding](#encoding) + - [Types of messages](#types-of-messages) + - [Message](#message) + - [Payload](#payload) + - [Payload](#payload-1) + - [Content types](#content-types) + - [Sticker content type](#sticker-content-type) + - [Message types](#message-types) + - [Clock vs Timestamp and message ordering](#clock-vs-timestamp-and-message-ordering) + - [Chats](#chats) + - [Contact Update](#contact-update) + - [Payload](#payload-2) + - [Contact update](#contact-update-1) + - [SyncInstallationContact](#syncinstallationcontact) + - [Payload](#payload-3) + - [SyncInstallationPublicChat](#syncinstallationpublicchat) + - [Payload](#payload-4) + - [PairInstallation](#pairinstallation) + - [Payload](#payload-5) + - [MembershipUpdateMessage and MembershipUpdateEvent](#membershipupdatemessage-and-membershipupdateevent) + - [Upgradability](#upgradability) + - [Security Considerations](#security-considerations) + - [Design rationale](#design-rationale) + +## Introduction + +In this document we describe the payload format and some special considerations. + +## Payload wrapper + +All payloads are wrapped in a [protobuf record](https://developers.google.com/protocol-buffers/) +record: + +```protobuf +message StatusProtocolMessage { + bytes signature = 4001; + bytes payload = 4002; +} +``` + +`signature` is the bytes of the signed `SHA3-256` of the payload, signed with the key of the author of the message. +The signature is needed to validate authorship of the message, so that the message can be relayed to third parties. +If a signature is not present but an author is provided by a layer below, the message is not to be relayed to third parties and it is considered plausibly deniable. + +## Encoding + +The payload is encoded using [Protobuf](https://developers.google.com/protocol-buffers) + +## Types of messages + +### Message + +The type `ChatMessage` represents a chat message exchanged between clients. + +#### Payload + +The protobuf description is: + +```protobuf +message ChatMessage { + // Lamport timestamp of the chat message + uint64 clock = 1; + // Unix timestamps in milliseconds, currently not used as we use whisper as more reliable, but here + // so that we don't rely on it + uint64 timestamp = 2; + // Text of the message + string text = 3; + // Id of the message that we are replying to + string response_to = 4; + // Ens name of the sender + string ens_name = 5; + // Chat id, this field is symmetric for public-chats and private group chats, + // but asymmetric in case of one-to-ones, as the sender will use the chat-id + // of the received, while the receiver will use the chat-id of the sender. + // Probably should be the concatenation of sender-pk & receiver-pk in alphabetical order + string chat_id = 6; + + // The type of message (public/one-to-one/private-group-chat) + MessageType message_type = 7; + // The type of the content of the message + ContentType content_type = 8; + + oneof payload { + StickerMessage sticker = 9; + } + + enum MessageType { + UNKNOWN_MESSAGE_TYPE = 0; + ONE_TO_ONE = 1; + PUBLIC_GROUP = 2; + PRIVATE_GROUP = 3; + // Only local + SYSTEM_MESSAGE_PRIVATE_GROUP = 4;} + enum ContentType { + UNKNOWN_CONTENT_TYPE = 0; + TEXT_PLAIN = 1; + STICKER = 2; + STATUS = 3; + EMOJI = 4; + TRANSACTION_COMMAND = 5; + // Only local + SYSTEM_MESSAGE_CONTENT_PRIVATE_GROUP = 6; + } +} +``` + +#### Payload + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | clock | `uint64` | The clock of the chat| +| 2 | timestamp | `uint64` | The sender timestamp at message creation | +| 3 | text | `string` | The content of the message | +| 4 | response_to | `string` | The ID of the message replied to | +| 5 | ens_name | `string` | The ENS name of the user sending the message | +| 6 | chat_id | `string` | The local ID of the chat the message is sent to | +| 7 | message_type | `MessageType` | The type of message, different for one-to-one, public or group chats | +| 8 | content_type | `ContentType` | The type of the content of the message | +| 9 | payload | `Sticker|nil` | The payload of the message based on the content type | + +#### Content types + +Content types are required for a proper interpretation of incoming messages. Not each message is plain text but may carry a different information. + +The following content types MUST be supported: +* `TEXT_PLAIN` identifies a message which content is a plaintext. + +There are also other content types that MAY be implemented by the client: +* `STICKER` +* `STATUS` +* `EMOJI` +* `TRANSACTION_COMMAND` + +##### Sticker content type + +A `ChatMessage` with `STICKER` `Content/Type` MUST also specify the ID of the `Pack` and +the `Hash` of the pack, in the `Sticker` field of `ChatMessage` + +```protobuf +message StickerMessage { + string hash = 1; + int32 pack = 2; +} +``` + +#### Message types + +Message types are required to decide how a particular message is encrypted and what metadata needs to be attached when passing a message to the transport layer. For more on this, see [3/WHISPER-USAGE](https://specs.status.im/spec/3). + + + + +The following messages types MUST be supported: +* `ONE_TO_ONE` is a message to the public group +* `PUBLIC_GROUP` is a private message +* `PRIVATE_GROUP` is a message to the private group. + +#### Clock vs Timestamp and message ordering + +If a user sends a new message before the messages sent while the user was offline are received, the new +message is supposed to be displayed last in a chat. This is where the basic algorithm of Lamport timestamp would fall short +as it's only meant to order causally related events. + +The status client therefore makes a "bid", speculating that it will beat the current chat-timestamp, s.t. the status client's +Lamport timestamp format is: `clock = `max({timestamp}, chat_clock + 1)` + +This will satisfy the Lamport requirement, namely: a -> b then T(a) < T(b) + +`timestamp` MUST be Unix time calculated when the message is created in milliseconds. This field SHOULD not be relied upon for message ordering. + +`clock` SHOULD be calculated using the algorithm of [Lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps). When there are messages available in a chat, `clock`'s value is calculated based on the last received message in a particular chat: `max(timeNowInMs, last-message-clock-value + 1)`. If there are no messages, `clock` is initialized with `timestamp`'s value. + +Messages with a `clock` greater than `120` seconds over the whisper timestamp SHOULD be discarded, in order to avoid malicious users to increase the `clock` of a chat arbitrarily. + +Messages with a `clock` less than `120` seconds under the whisper timestamp might indicate an attempt to insert messages in the chat history which is not distinguishable from a `datasync` layer re-transit event. A client MAY mark this messages with a warning to the user, or discard them. + +`clock` value is used for the message ordering. Due to the used algorithm and distributed nature of the system, we achieve casual ordering which might produce counterintuitive results in some edge cases. For example, when one joins a public chat and sends a message before receiving the exist messages, their message `clock` value might be lower and the message will end up in the past when the historical messages are fetched. + +#### Chats + +Chat is a structure that helps organize messages. It's usually desired to display messages only from a single recipient or a group of recipients at a time and chats help to achieve that. + +All incoming messages can be matched against a chat. Below you can find a table that describes how to calculate a chat ID for each message type. + +|Message Type|Chat ID Calculation|Direction|Comment| +|------------|-------------------|---------|-------| +|PUBLIC_GROUP|chat ID is equal to a public channel name; it should equal `chatId` from the message|Incoming/Outgoing|| +|ONE_TO_ONE|let `P` be a public key of the recipient; `hex-encode(P)` is a chat ID; use it as `chatId` value in the message|Outgoing|| +|user-message|let `P` be a public key of message's signature; `hex-encode(P)` is a chat ID; discard `chat-id` from message|Incoming|if there is no matched chat, it might be the first message from public key `P`; you can discard it or create a new chat; Status official clients create a new chat| +|PRIVATE_GROUP|use `chatId` from the message|Incoming/Outgoing|find an existing chat by `chatId`; if none is found, we are not a member of that chat or we haven't joined that chat, the message MUST be discarded | + +### Contact Update + +`ContactUpdate` is a message exchange to notify peers that either the +user has been added as a contact, or that information about the sending user have +changed. + +```protobuf +message ContactUpdate { + uint64 clock = 1; + string ens_name = 2; + string profile_image = 3; +} +``` + +#### Payload + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | clock | `uint64` | The clock of the chat with the user | +| 2 | ens_name | `string` | The ENS name if set | +| 3 | profile_image | `string` | The base64 encoded profile picture of the user | + +#### Contact update + +A client SHOULD send a `ContactUpdate` to all the contacts each time: + +- The ens_name has changed +- The profile image is edited + +A client SHOULD also periodically send a `ContactUpdate` to all the contacts, the interval is up to the client, the Status official client sends these updates every 48 hours. + +### SyncInstallationContact + +`SyncInstallationContact` messages are used to synchronize in a best-effort the contacts to other devices. + +```protobuf +message SyncInstallationContact { + uint64 clock = 1; + string id = 2; + string profile_image = 3; + string ens_name = 4; + uint64 last_updated = 5; + repeated string system_tags = 6; +} +``` + + +#### Payload + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | clock | `uint64` | clock value of the chat | +| 2 | id | `string` | id of the contact synced | +| 3 | profile_image | `string` | `base64` encoded profile picture of the user | +| 4 | ens_name | `string` | ENS name of the contact | +| 5 | `array[string]` | Array of `system_tags` for the user, this can currently be: `":contact/added", ":contact/blocked", ":contact/request-received"`| + +### SyncInstallationPublicChat + +`SyncInstallationPublicChat` message is used to synchronize in a best-effort the public chats to other devices. + +```protobuf +message SyncInstallationPublicChat { + uint64 clock = 1; + string id = 2; +} +``` + +#### Payload + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | clock | `uint64` | clock value of the chat | +| 2 | id | `string` | id of the chat synced | + +### PairInstallation + +`PairInstallation` messages are used to propagate informations about a device to its paired devices. + +```protobuf +message PairInstallation { + uint64 clock = 1; + string installation_id = 2; + string device_type = 3; + string name = 4; +} +``` + +#### Payload + +| Field | Name | Type | Description | +| ----- | ---- | ---- | ---- | +| 1 | clock | `uint64` | clock value of the chat | +| 2| installation_id | `string` | A randomly generated id that identifies this device | +| 3 | device_type | `string` | The OS of the device `ios`,`android` or `desktop` | +| 4 | name | `string` | The self-assigned name of the device | + +### MembershipUpdateMessage and MembershipUpdateEvent + +`MembershipUpdateEvent` is a message used to propagate information about group membership changes in a group chat. +The details are in the [Group chats specs](status-group-chats-spec.md) + +## Upgradability + +There are two ways to upgrade the protocol without breaking compatibility: + +- Accretion is always supported +- Deletion of existing fields or messages is not supported and might break compatibility + +## Security Considerations + +- + +## Design rationale From ca530d9d80681e9d6442fd10cac3e0f2607d88b7 Mon Sep 17 00:00:00 2001 From: Andrea Maria Piana Date: Thu, 30 Apr 2020 12:25:37 +0200 Subject: [PATCH 2/3] Add image content-type --- docs/draft/6-payloads.md | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/docs/draft/6-payloads.md b/docs/draft/6-payloads.md index 20331d9..d4756b0 100644 --- a/docs/draft/6-payloads.md +++ b/docs/draft/6-payloads.md @@ -1,14 +1,14 @@ --- permalink: /spec/6 -parent: Stable specs +parent: Draft specs title: 6/PAYLOADS --- # 6/PAYLOADS -> Version: 0.2 +> Version: 0.3 > -> Status: Stable +> Status: Draft > > Authors: Adam Babik , Andrea Maria Piana , Oskar Thorén (alphabetical order) @@ -35,6 +35,7 @@ as various clients created using different technologies. - [Payload](#payload-1) - [Content types](#content-types) - [Sticker content type](#sticker-content-type) + - [Image content type](#image-content-type) - [Message types](#message-types) - [Clock vs Timestamp and message ordering](#clock-vs-timestamp-and-message-ordering) - [Chats](#chats) @@ -112,6 +113,7 @@ message ChatMessage { oneof payload { StickerMessage sticker = 9; + ImageMessage image = 10; } enum MessageType { @@ -130,6 +132,7 @@ message ChatMessage { TRANSACTION_COMMAND = 5; // Only local SYSTEM_MESSAGE_CONTENT_PRIVATE_GROUP = 6; + IMAGE = 7; } } ``` @@ -146,7 +149,7 @@ message ChatMessage { | 6 | chat_id | `string` | The local ID of the chat the message is sent to | | 7 | message_type | `MessageType` | The type of message, different for one-to-one, public or group chats | | 8 | content_type | `ContentType` | The type of the content of the message | -| 9 | payload | `Sticker|nil` | The payload of the message based on the content type | +| 9 | payload | `Sticker|Image|nil` | The payload of the message based on the content type | #### Content types @@ -160,6 +163,7 @@ There are also other content types that MAY be implemented by the client: * `STATUS` * `EMOJI` * `TRANSACTION_COMMAND` +* `IMAGE` ##### Sticker content type @@ -173,6 +177,26 @@ message StickerMessage { } ``` +##### Image content type + +A `ChatMessage` with `IMAGE` `Content/Type` MUST also specify the `payload` of the image +and the `type` + +```protobuf +message ImageMessage { + bytes payload = 1; + ImageType type = 2; +} + +enum ImageType { + UNKNOWN_IMAGE_TYPE = 0; + IMAGE_JPEG = 1; + IMAGE_PNG = 2; + IMAGE_WEBP = 3; +} + +``` + #### Message types Message types are required to decide how a particular message is encrypted and what metadata needs to be attached when passing a message to the transport layer. For more on this, see [3/WHISPER-USAGE](https://specs.status.im/spec/3). From 8f7746955510918c5e12ddd16986ce476356c0ae Mon Sep 17 00:00:00 2001 From: Andrea Maria Piana Date: Fri, 15 May 2020 09:17:17 +0200 Subject: [PATCH 3/3] Address feedback --- docs/draft/6-payloads.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/draft/6-payloads.md b/docs/draft/6-payloads.md index d4756b0..4ffc87e 100644 --- a/docs/draft/6-payloads.md +++ b/docs/draft/6-payloads.md @@ -100,10 +100,7 @@ message ChatMessage { string response_to = 4; // Ens name of the sender string ens_name = 5; - // Chat id, this field is symmetric for public-chats and private group chats, - // but asymmetric in case of one-to-ones, as the sender will use the chat-id - // of the received, while the receiver will use the chat-id of the sender. - // Probably should be the concatenation of sender-pk & receiver-pk in alphabetical order + // Chat id is the ID of the chat string chat_id = 6; // The type of message (public/one-to-one/private-group-chat) @@ -186,15 +183,14 @@ and the `type` message ImageMessage { bytes payload = 1; ImageType type = 2; + enum ImageType { + UNKNOWN_IMAGE_TYPE = 0; + PNG = 1; + JPEG = 2; + WEBP = 3; + GIF = 4; + } } - -enum ImageType { - UNKNOWN_IMAGE_TYPE = 0; - IMAGE_JPEG = 1; - IMAGE_PNG = 2; - IMAGE_WEBP = 3; -} - ``` #### Message types @@ -357,3 +353,8 @@ There are two ways to upgrade the protocol without breaking compatibility: - ## Design rationale + + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).