Commit Graph

109 Commits

Author SHA1 Message Date
frank b6d4e75cf1
Feat/sync activity center notification (#3535)
fix flaky test: TestRetrieveBlockedContact

resolve conflict when rebase origin/develop

Feat/sync activity center notification (#3581)

* feat: sync activity center notification

* add test

* fix lint issue

* fix failed test

* addressed feedback from sale

* fix failed test

* addressed feedback from ilmotta

go generate ./protocol/migrations/sqlite/...

feat: add updated_at for syncing activity center notification
2023-06-10 10:00:17 +08:00
Alex Jbanca e8c4b7647f
chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5 (#3559)
* chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5

Changes:
### github.com/mutecomm/go-sqlcipher
1. The improved crypto argorighms from go-sqlcipher v3 are merged in v4
	Tags:
	v4.4.2-status.1 - merge `burn_stack` improvement
	v4.4.2-status.2 - merge `SHA1` improvement
	v4.4.2-status.4- merge 'AES' improvement
2. Fixed `go-sqlcipher` to support v3 database in compatibility mode (`sqlcipher` already supports this) (Tag: v4.4.2-status.3)
3. Upgrade `sqlcipher` to v5.4.5 (Tag: v4.5.4-status.1)

### github.com/status-im/migrate/v4
1. Upgrade `go-sqlcipher` version in `github.com/status-im/migrate/v4`

### status-go
1. Upgrade `go-sqlcipher` and `migrate` modules in status-go
2. Configure the DB connections to open the DB in v3 compatibility mode

* chore(upgradeSQLCipher): Use sqlcipher v3 configuration to encrypt a plain text database

* chore(upgradeSQLCipher): Scanning NULL BLOB value should return nil

Fixing failing tests: TestSyncDeviceSuite/TestPairingSyncDeviceClientAsReceiver;  TestSyncDeviceSuite/TestPairingSyncDeviceClientAsSender

Considering the following configuration:
1. Table with BLOB column has 1 NULL value
2. Query the value
3. Rows.Scan(&dest sql.NullString)

Expected: dest.Valid == false; dest.String == nil
Actual: dest.Valid == true; dest.String == ""

* chore: Bump go-sqlcipher version to include NULL BLOB fix
2023-06-07 08:58:01 +03:00
Jonathan Rainville 3f00869e1f
fix(contacts): add contact to ban and unban responses (#3558) 2023-06-01 13:26:12 -04:00
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
Jonathan Rainville 34127cd14f
fix(messenger_handler): fix checkForDeletes logic with images (#3477) (#3488) 2023-05-16 11:00:49 -04:00
John Ngei 9dea0aaee3
Remove contact from the contact list of contacts the blocked contact
* Remove contact from the user's list of contacts when the contact is blocked by the user

* added test for removed blocked contact
2023-04-20 13:52:08 +03:00
Jonathan Rainville 62da2d404c
fix(image-album): make sure to delete all images part of an album (#3380)
* fix(image-album): make sure to delete all images part of an album

* test(delete): add test that deletes a message part of an album

* test(delete): add test where the signal to delete images is after

also adds the handling of deleteForMe

* fix(delete): add album deletion handling for deleteForMe
2023-04-14 13:17:56 -04:00
Andrea Maria Piana 5d0e08ec7b Add spiff workflow cmd 2023-03-16 14:31:34 +00:00
Andrea Maria Piana 44a0f5b74d
Add collapsed community categories 2023-03-14 17:13:21 +00:00
Andrea Maria Piana b69042e7d7 Fix duplicated notifications on receiving contact update
This commit fixes an issue where when accepting a contact request
the other end would display an extra notification.

It also changes WaitOnResponse to collect results. This should make
tests less flaky, since sometimes messages are processed in different
batches.
Now we need to be though exact on what we expect from the response (i.e
use == instead of >, otherwise the same behavior applies)
This uncovered a couple of issues with messenger.Merge, so I have moved
the struct to use a map based collection instead of an array.
2023-02-14 09:22:37 +00:00
Andrea Maria Piana dea21f440a Linting and some fixes on contact requests handling 2023-02-08 17:48:09 +00:00
Andrea Maria Piana cefa0089dc
Split clock values for contacts 2023-01-31 15:43:00 +00:00
Andrea Maria Piana 7e1a894ab8 Add compressed key to multiaccount/messages/contacts 2023-01-30 16:40:40 +00:00
Patryk Osmaczko b5887c9c60 chore: extend `TestRetrieveBlockedContact`
Extended test with unblock scenario.
2022-11-11 15:44:22 +01:00
Stefan b9d6e6d8c8 feat: sync saved addresses
Main changes:
- Extend saved addresses DB with sync info: sync timestamp, update timestamp
and soft removed flag
- Create custom protobuf message payload to sync saved addresses
- Cleanup saved addresses on each start of messenger, by deleting
soft removed older entries
- Sync all saved addresses on Messenger.SyncDevices calls
- Sync particular changes to saved addresses
- Add SavedAddressManager instance to messenger
  - Note, can't find a clean way to pass the SavedAddressManager to the
  messenger, so we create another one
- Add tests for sync and new DB API

Closes: #7229
2022-10-10 11:26:35 +02:00
Michał 057c7915f4
chore: use provided pictures visibility options when creating settings db entry (#2867)
When initially creating settings, properties 'initialize profile_pictures_show_to'
and 'profile_pictures_visibility' are set according to the provided Setting object.
2022-10-05 12:53:42 +02:00
Richard Ramos bf9abfc350
feat: make kdf iterations configurable when creating a multiaccount (#2796) 2022-09-27 16:27:20 -04:00
Patryk Osmaczko c01316524c feat: make any member able to add new users to group chat
closes: #2823, #2825
2022-09-13 12:16:14 +02:00
Andrea Maria Piana 09ba88c19d Join group on accepting activity center notification 2022-06-08 11:48:45 +01:00
Patryk Osmaczko 5df5d6b519 feat: remove explicit group chat join
relates: #status-im/status-desktop/5717
2022-06-08 11:48:45 +01:00
Andrea Maria Piana 1bfde4c4cc Initial support for mutual contact requests 2022-05-31 09:12:36 +01:00
Samuel Hawksby-Robinson e67592d556
Sync Settings (#2478)
* Sync Settings

* Added valueHandlers and Database singleton

Some issues remain, need a way to comparing incoming sql.DB to check if the connection is to a different file or not. Maybe make singleton instance per filename

* Added functionality to check the sqlite filename

* Refactor of Database.SaveSyncSettings to be used as a handler

* Implemented inteface for setting sync protobuf factories

* Refactored and completed adhoc send setting sync

* Tidying up

* Immutability refactor

* Refactor settings into dedicated package

* Breakout structs

* Tidy up

* Refactor of bulk settings sync

* Bug fixes

* Addressing feedback

* Fix code dropped during rebase

* Fix for db closed

* Fix for node config related crashes

* Provisional fix for type assertion - issue 2

* Adding robust type assertion checks

* Partial fix for null literal db storage and json encoding

* Fix for passively handling nil sql.DB, and checking if elem has len and if len is 0

* Added test for preferred name behaviour

* Adding saved sync settings to MessengerResponse

* Completed granular initial sync and clock from network on save

* add Settings to isEmpty

* Refactor of protobufs, partially done

* Added syncSetting receiver handling, some bug fixes

* Fix for sticker packs

* Implement inactive flag on sync protobuf factory

* Refactor of types and structs

* Added SettingField.CanSync functionality

* Addressing rebase artifact

* Refactor of Setting SELECT queries

* Refactor of string return queries

* VERSION bump and migration index bump

* Deactiveate Sync Settings

* Deactiveated preferred_name and send_status_updates

Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
2022-03-23 18:47:00 +00:00
Richard Ramos 50ec6f97e0 feat: display name 2022-03-14 13:48:34 -04:00
frank 4cc53630d5 Sync bookmarks 2022-02-17 12:40:33 +00:00
Samuel Hawksby-Robinson bd3f0f436b
Turning the tests back on (#2519) 2022-02-03 20:37:41 +00:00
Richard Ramos 98784b752a
feat: desktop mailserver cycle (#2481) 2022-01-12 12:02:01 -04:00
Roman Volosovskyi 3564630c33
Allow sending messages offline (non-datasync messages) 2021-12-07 12:12:49 +02:00
Parvesh Monu 83c3849899
Decline pending group invites from user, when user is blocked (#2455) 2021-12-06 18:14:40 +05:30
Vitaliy Vlasov f271277b76 Disable failing tests
Fix lint
2021-11-30 20:52:47 +02:00
Andrea Maria Piana 6724cf4c75 Block contact endpoint 2021-11-15 18:53:35 +00:00
Andrea Maria Piana ca6246d5f2 Set last backup 2021-11-15 18:53:35 +00:00
Andrea Maria Piana be1a261d9b Check for verified records when creating a contact 2021-11-15 18:53:35 +00:00
Andrea Maria Piana a7b0c6c933 Backup removed & added by them contacts 2021-11-15 18:53:35 +00:00
Anthony Laibe c95c29bba5 feat: Push messages to telemetry server when receiving messages 2021-11-10 09:34:14 +01:00
Parvesh Monu c20e8ebdef
Store Highlight field for identify new chats (#2384) 2021-10-21 22:34:56 +05:30
Andrea Maria Piana 438c2e8883 contacts performance 2021-10-04 12:19:15 +02:00
Samuel Hawksby-Robinson d9ea6a910e Tests and linting 2021-09-30 13:02:41 +01:00
Samuel Hawksby-Robinson 0ff7a864ec Attempting to resolved 'no such column: profile_pictures_show_to'
Have resolved the settings table not found but now the column is not migrating for some reason
2021-09-30 13:02:41 +01:00
Samuel Hawksby-Robinson a50048e290 Updated tests to match refactor 2021-09-30 13:02:41 +01:00
Samuel Hawksby-Robinson 33487d64de Updated tests to include tests on IdentityImage.Encrypted state 2021-09-30 13:02:41 +01:00
Samuel Hawksby-Robinson 1534fc15c9 Updated contact key gen in test so index call wouldn't be ambiguous 2021-09-30 13:02:41 +01:00
Samuel Hawksby-Robinson b009bac5fb Switched encryption of payload and keys to AES
Added Encrypted field to the protobuf
2021-09-30 13:02:41 +01:00
Andrea Maria Piana bd1e6b0033 Fix handling on public chat messages 2021-09-15 09:28:51 +01:00
Roman Volosovskyi 0c0e02e93a
Return counter for messages with mention which are marked as seen 2021-08-31 11:49:40 +03:00
Jonathan Rainville cdc7c55030
Delete messages (#2279)
* feat: add api and handling to delete messages

* fix(delete): add whitelist to message types that can be deleted
2021-07-26 17:06:32 -04:00
Andrea Maria Piana d84da4f515 Fix some tests and restore private/public RPC client 2021-07-20 10:57:38 +02:00
Bitgamma 58e39adf03
add ChatID to transaction commands (#2274) 2021-07-07 14:18:18 +03:00
frank c799e5eb6b Drop audio messages in public chats 2021-07-02 13:09:30 +02:00
Andrea Maria Piana 075ee9c29f Merge message handler with messenger 2021-06-29 13:15:15 +02:00
Andrea Maria Piana 5d856adc0c Change to map in messenger response 2021-06-29 13:15:15 +02:00