Commit Graph

59 Commits

Author SHA1 Message Date
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
Andrea Maria Piana 5d81866957 Add request to response when canceling request 2023-03-14 13:50:18 +00:00
Siddarth Kumar 6ac2308ee1
Re-organise the code to be more modular (#3172) 2023-03-01 17:23:17 +05:30
Andrea Maria Piana 1c660c3525 Move images to shared namespace 2023-02-02 19:18:06 +00:00
Pascal Precht 2cac37c1fb fix(images): ensure decode from URL emits error on HTTP error code
We should emit an error when the request to an image to be fetched
returns and HTTP error code. Otherwise, we'll run into other higher
level errors down the line, which are misleading

Example: I kept seeing "image content type not supported" errors,
although the content type *is* supported. The actual problem was that
the decode function operates on non existing image bytes.
2022-09-19 12:55:17 +02:00
frank 6e4ac1c495 Use image server for all kind of images and integrate identity rings 2022-09-16 09:52:08 +01:00
frank 325efd01d0 Implement identity ring 2022-09-16 09:52:08 +01:00
Samuel Hawksby-Robinson 41cb9d7bea Added httptest.NewServer to DecodeFromURL and CryptoOnRamp tests 2022-09-09 15:07:48 +01:00
Samuel Hawksby-Robinson 399090a44b Added fix for pinging a remote service in tests 2022-09-09 15:07:48 +01:00
Samuel Hawksby-Robinson cea3b3db0d Added FileSizeError instead of clunky awful string comparison 2022-09-05 16:06:04 +01:00
Samuel Hawksby-Robinson 45b287370a Implemented more comprehensive file compression to handle large files 2022-09-05 16:06:04 +01:00
Michal Iskierko e3f451cd79 feat: Handle image in group chat
Add image_payload column to chats table.
Add Base64Image to chat struct.
Add ImageChange event to propagate image.
Change EditChat API - use CroppedImage.
Process and crop image in EditGroupChat.
2022-09-01 12:19:01 +02:00
Samuel Hawksby-Robinson 344272ee08 Moved protobuf marshalling closer to the structs that get marshalled 2022-08-22 13:27:59 +01:00
Samuel Hawksby-Robinson 06f4b85792 undoing the rookie things I did 2022-08-22 13:27:59 +01:00
Samuel Hawksby-Robinson 344458d74a Added multiaccounts.Database tests for new funcs 2022-08-22 13:27:59 +01:00
Samuel Hawksby-Robinson 38c8e8a1fe Added payload marshal and unmarshal to/from protobuf 2022-08-22 13:27:59 +01:00
Jakub Sokołowski 530f3c7a3a
rename status-react to status-mobile
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2022-07-17 15:37:14 +02:00
Stefan 63e58ba035 feat: add banner support for communities
Add banner image as a special `IdentityImage` beside "thumbnail" and "large"

Banner input cropped image processing

- Resize to keep in the limits of `BannerDim`
- Encode to match the file size limits define for banner
- Don't scale up. This can be done efficiently in the UI

Changes to `images` module

- Refactor `EncodeToBestSize` as `EncodeToLimits` to accept arbitrary dimensions
  and allow for custom size
- Define `DimensionLimits` for banner not to exceed 450 KB and a rough estimate
  for the ideal size
2022-05-27 14:42:14 +03:00
Vitaliy Vlasov 16197dc807 Sync profile picture 2022-04-21 19:50:31 +03:00
Shivek Khurana a6e7ff6ddd
🖼 Set any url as profile picture (useful for NFTs) (#2367)
* Sound check
* Add DecodeImageURL fn
* Introduce crop dynamics
* Add center crop calculations png

* Apply suggestions from code review
Co-authored-by: RichΛrd <info@richardramos.me>

* Fix lint
* Rebase and update version
Co-authored-by: RichΛrd <info@richardramos.me>
2021-09-21 14:30:44 +05:30
Andrea Maria Piana 58f2dac673 Remove image log 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson ca80140c1a Resolved broken tests 2020-12-17 14:10:00 +01:00
Andrea Maria Piana 1adfa38611 Check hash in shouldPublish 2020-12-17 14:10:00 +01:00
Andrea Maria Piana 618524d8cb debug 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 79f2b0bcc7 Address consistent json field case convention - use camelCase 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 5aa8492315 Fix failing tests 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson a77a40fdf4 Resolved linting issues 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 2d72d21498 Rename file now that db logic has been moved 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 193ab30ada Refactor injecting multiaccounts into messenger and service 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 7b230d5c9d Updated message handling from public and private chat 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 8be391fc79 Bug fix for nil pointer on image save 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 9545f8e306 Added log to debug an image store nil pointer panic 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson a564066c4f Added public chat ChatIdentity publish 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 48a34072bb Added ChatIdentity attachment to ContactCodeAdvert 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson dcee7f8de7 Changed identity_images column name to name 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 2960fc49d4 Implemented GetImages and StoreImages in API 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson a7358fcd3f images package tidy up 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 7ea153bae1 Added Identity Image Database delete 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson daa5ca1dbd Added GetIdentityImage by image name 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson aad761b24b Added IdentityImage Database store and get tests 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson df55822e29 Added StoreIdentityImages to Database 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 01657b4665 Added IdentityImage JSON marshalling tests 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 7beebaaf92 Added images/database functionality 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 1b59d088d8 Integrated GetMimeType() 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 8ec0ec4341 Added TestEncodeToBestSize() 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 3b7fbf94d3 Added SaveProfileImage endpoint
Also added EncodeToBestSize func
2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 9bc240fc71 Improved Resize() and added tests 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson a6df8adeba Renamed Decoding and Encoding funcs 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 0d6e6699ec Split out encode code from decode, same for tests 2020-12-17 14:10:00 +01:00
Samuel Hawksby-Robinson 36105aaac0 Split out encode code from decode, same for tests 2020-12-17 14:10:00 +01:00