Commit Graph

113 Commits

Author SHA1 Message Date
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
Volodymyr Kozieiev a669e7d038
More fields added to ChatPreview. LastMessage now can be a deleted message (#3397) 2023-05-08 18:02:54 +01:00
Jonathan Rainville 79cbe6a410
fix(mentions): deleting or editing a mention should remove the mention (#3421)
* fix(mentions): deleting or editing a mention should remove the mention

* test(edit): add a test for mentions in edits

* test(delete): add test for deleting a message with a mention
2023-04-27 10:22:26 -04:00
Eng Zer Jun 83ad76637a
test: use `T.TempDir` to create temporary test directory (#2746)
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-04-26 21:39:51 +01:00
Andrea Maria Piana 7650f3003e Fix some issues with pinned messages
There were a couple of issues on how we handle pinned messages:

1) Clock of the message was only checked when saving, meaning that the
   client would receive potentially updates that were not to be
   processed.
2) We relied on the client to generate a notification for a pinned
   message by sending a normal message through the wire. This PR changes
   the behavior so that the notification is generated locally, either on
   response to a network event or client event.
3) When deleting a message, we pull all the replies/pinned notifications
   and send them over to the client so they know that those messages
   needs updating.
2023-04-25 16:02:48 +01:00
Ibrahem Khalil d0766e3023
Update album_images_count column on receiver side (#3375) 2023-04-05 15:24:55 +02:00
Ibrahem Khalil bd593373b2
Add album count key to messages (#3347) 2023-03-30 12:02:20 +02:00
Boris Melnik 6666110ba4
feat(images): Add albumId to images album and allow to receive image messages without a text (#3287) 2023-03-23 19:06:50 +03:00
Vitaliy Vlasov 27a76f43ba Compute emojiHash on-the-fly 2023-03-06 13:22:40 +02:00
Andrea Maria Piana 11db9f2119
Add test for everyone tag & fix migration order
A migration was added out-of-order, which meant that in clients who
had already run the migration after, it would be skip.
This commit re-adds the migration so it's run, tested against an empty
account and one that had already migrated.
2023-02-24 10:18:26 +00:00
Andrea Maria Piana 1c660c3525 Move images to shared namespace 2023-02-02 19:18:06 +00:00
Omar Basem 2485143e20
Fix image fields (#3121)
* fix: image fields
2023-02-02 12:00:49 +04:00
yqrashawn d4ec412df2
feat: add deleted by xxx support (#3077) 2023-02-01 08:57:35 +08:00
Andrea Maria Piana 7e1a894ab8 Add compressed key to multiaccount/messages/contacts 2023-01-30 16:40:40 +00:00
Pascal Precht cedc1a5fd1 fix: add discord message data to quoted messages
Because `QuotedMessage` doesn't include imported message data,
some of the author information in imported messages is lost in
frontends.

This commit adds a `discordMessage` (soon replaced by `importedMessage`)
to `Quotedmessage`, although only hydrated with a subset of data,
namely author display name and avatar URL, as those are the only ones
needed by front-end atm.
2023-01-17 12:08:42 +01:00
Jonathan Rainville cb1d80c082 feat(QuotedMessage): pass Deleted to QuotedMessage
This is to let the front end determine a new message if the quoted message is deleted instead of just unknown
2023-01-12 11:47:38 -05:00
Omar Basem fe9996d95f
Image width height (#3061)
* feat: add image width & height
2023-01-12 13:43:14 +04:00
Jonathan Rainville c4bf60b815 feat: make replies act as mentions
and add a reply test to the messenger
2023-01-10 13:39:57 -05:00
Jakub Sokołowski cceba787fa
replace deprecated ioutil.TempDir with os.MkdirTemp
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-01-03 15:40:11 +01:00
Pascal Precht e7d827fff1 feat: recgonize `0x00001` as system mention
This "id" or "tag" will be interpeted/interpolated as `@everyone` by
clients.

For this change to work

https://github.com/status-im/markdown/pull/5

is needed.
2022-12-20 15:13:22 +01:00
Jonathan Rainville 4a0eb56574 fix: fix special messages not being received because not encrypted 2022-12-15 10:48:41 -05:00
Omar Basem d03691c2f2
Images Album (#3021)
* feat: add `AlbumID` field to messages
2022-12-14 16:25:45 +04:00
Mikhail Rogachev cd9fb48579
feat(ActivityCenter): Add message for canceling contact requests (#2992)
* feat(ActivityCenter): Add message for canceling contact requests

* feat(Contacts): GetVerificationRequestSentTo returns last verification request

* Fix error message in protocol/messenger_contact_verification.go

Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>

Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
2022-12-14 12:27:02 +04:00
Samuel Hawksby-Robinson 3e8eed5faf Added device name set on Messenger.Start() 2022-11-25 11:33:58 +00:00
Andrea Maria Piana 9b670ff453 Correctly abridge events after changes in joined users 2022-11-18 11:43:23 +00:00
Andrea Maria Piana 2572321063 Add notification for trustworthy/untrustworthy 2022-11-16 16:17:13 +00:00
Richard Ramos 0f7c9f52d8 feat: mark automatic status updates as ephemeral 2022-11-09 15:49:26 +03:00
Pascal Precht caa20e616e feat(communities): handle markdown in imported discord messages 2022-10-28 20:03:39 +02:00
Andrea Maria Piana de61ed1213 Handle identity verifications 2022-10-26 17:19:44 +01:00
Andrea Maria Piana a89f4b2d71 Add read/unread messenger functions for activity center 2022-10-26 17:19:44 +01:00
Andrea Maria Piana bef211b5d6 Send all encryption keys 2022-10-20 12:19:44 +01:00
Omar Basem 938e7987ca
feat: edit/delete/reply for emoji/sticker/audio (#2771)
* feat: edit/delete/reply for emoji/sticker/audio
2022-10-05 21:54:47 +04:00
Pascal Precht b01a861e8e feat: add `DiscordMessageAttachment` types and APIs
This adds a new `DiscordMessageAttachment` type which is part of
`DiscordMessage`. Along with that type, there's also a new database
table for `discord_message_attachments` and corresponding persistence
APIs.

This commit also changes how chat messages are retrieved.
Here's why:

`DiscordMessage` can have multiple `DiscordMessageAttachment`.
A chat message can have a `DiscordMessage`.

Because we're `LEFT JOIN`'ing the discord message attachments into the
chat messages, there's a possibility of multiple rows per message.

Hence, this commit ensures we collect queried discord message
attachments on chat messages.
2022-09-29 11:38:29 +02:00
yqrashawn f47cb8572d
feat: delete for me (#2866) 2022-09-28 19:42:17 +08:00
Richard Ramos bf9abfc350
feat: make kdf iterations configurable when creating a multiaccount (#2796) 2022-09-27 16:27:20 -04:00
Andrea Maria Piana a182f3e699 Fix bug with tracking & merge community changes 2022-09-16 22:45:29 +01:00
Andrea Maria Piana d689b337c9 Return ID and use chat topic when sending community message 2022-09-15 09:44:56 +01:00
Pascal Precht 081974da1e feat: add `discord_messages` table and persistence APIs
This adds a new `discord_messages` table and extends the persistence
APIs such that `MessagesByID` and `MessageByID` will return user
messages that include their discord message payload.

It also adds APIs to save individual discord messages.
2022-08-11 10:49:23 +02:00
Andrea Maria Piana 897bd0f58f Dont send notification to our own devices 2022-07-18 17:17:20 +01:00
Jakub Sokołowski 530f3c7a3a
rename status-react to status-mobile
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2022-07-17 15:37:14 +02:00
Samuel Hawksby-Robinson 366c088ec5 Fix cycle import and added private key to Client 2022-06-20 15:33:09 +01:00
Vitaliy Vlasov 4f722b6fe8 Communities encryption integration 2022-06-17 21:24:39 +03:00
Samuel Hawksby-Robinson 7f149f93c1
Get preferred network IP and refactor server package to increase reusability (#2626)
* Added function to get preffered network IP

Also done some refactor work oon server package to make a lot more reusable

* Added server.Option and simplified handler funcs

* Added serial number deterministically generated from pk

* Debugging TLS server connection

* Implemented configurable server ip

When accessing over the network the server needs to listen on the network port and not localhost or 127.0.0.1 . Also the cert can now have a dedicated IP

* Refactor of URL funcs to use the url package

* Removed redundant Options pattern in favour of config param

* Added full server test using GetOutboundIP

* Remove references and usage of Server.port

The application does not need to set the port, we rely on the net.Listener to pick a port.

* Version bump

* Added ToECDSA func and improved cert testing

* Added error check in test

* Split Server types, embedding raw Server funcs into specialised server types

* localhost

* Implemented DNS and IP based cert gen

ios doesn't allow for restricted ip addresses to be used in a valid tls cert

* Replace listener handling with original port store

Also added handlers as a parameter of the Server
2022-06-15 15:49:31 +01:00
Andrea Maria Piana 1bfde4c4cc Initial support for mutual contact requests 2022-05-31 09:12:36 +01:00
Richard Ramos 0048aaebcc
feat: add IPFS rate limiter for downloading stickers and use http server for retrieving stickers (#2611) 2022-05-09 09:07:57 -04:00
andrey 01b6988260 QuotedMessage image from http server 2022-03-21 10:33:08 +01:00
Richard Ramos 50ec6f97e0 feat: display name 2022-03-14 13:48:34 -04:00
andrey 598b83757c load audio from http server 2022-03-01 15:58:32 +01:00
flexsurfer 5925b3b7cc
http server for images (#2418)
Serve images over HTTPS

Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Co-authored-by: Michele Balistreri <michele@bitgamma.com>
2022-02-10 18:19:34 +01:00
Richard Ramos 98784b752a
feat: desktop mailserver cycle (#2481) 2022-01-12 12:02:01 -04:00