Commit Graph

1103 Commits

Author SHA1 Message Date
Khushboo-dev-cpp e1004b4aec
feat: Add new settings for include watch-only accounts (#3676) 2023-06-28 12:05:06 +02:00
Patryk Osmaczko a2a2e61163 refactor: extract communities_messenger_token_permissions_test
`communities_messenger_test` was unnecessarily polluted with passwords
and addresses.
2023-06-28 07:56:51 +02:00
Ibrahem Khalil fc353d5dff
Only send PNs to chats that are unmuted (#3673) 2023-06-27 19:19:21 +04:00
Pascal Precht b8b402da8d feat: store check channel permission responses
This commit adds new tables to the database and APIs in `Messenger` and
communities `Manager` to store `CheckChannelPermissionsResponse`s.

The responses are stored whenever channel permissions have been checked.

The reason we're doing this is so that clients can retrieve the last
known channel permission state before waiting for onchain checks to
finish.
2023-06-27 12:13:59 +02:00
Pascal Precht a8678575a7 fix: admins are not allowed to kick or ban other admins 2023-06-27 09:14:26 +02:00
Andrea Maria Piana efad2ef6a2 Add BeforeDispatch callback 2023-06-26 13:36:20 +01:00
Andrea Maria Piana 14e3eafaeb Handle async raw message confirmations
Sometimes confirmation for raw messages are received before the record
is actually saved in the database.

In this case, the code will preserve the Sent status.
2023-06-26 13:36:20 +01:00
Alexander 83238283b4
Updates ring width and palette to match the designs (#3671) 2023-06-26 13:17:59 +01:00
Patryk Osmaczko 783a6b9153 fix: remove rekeying unencrypted communities 2023-06-23 16:33:58 +02:00
MishkaRogachev ecf493e6cd chore: less flaky AccountManagerMock for community tests 2023-06-23 14:54:44 +01:00
Andrea Maria Piana 6e19098dee Run tests separately, make sure messenger is shutdown, fix torrent test 2023-06-23 11:30:04 +01:00
Jonathan Rainville 9c596343be
feat(communities): enable selecting addresses to pass when joining (#3656)
Improve `RequestToJoinCommunity`  to accept `Addresses` in the request. If `Addresses` is not empty, we then only pass to the owner the selected addresses. The others are ignored.
Does not validate that the addresses in the slice are part of the user's wallet. Those not part of the wallet are just ignored.
2023-06-22 14:59:07 -04:00
Boris Melnik 3e7d1a5f34
fix(migration): Fixes mute_till migration (#3653) 2023-06-22 18:49:58 +03:00
Ibrahem Khalil d3e650d5e5
Make chat muting only works on one chat and not to completely block PNs from a user (#3647) 2023-06-22 08:06:32 +03:00
Mikhail Rogachev 2fb87b5f0d
Chore: improve mocking for community tests (#3637)
* chore: different wallet addresses for different users in community tests

* chore: fix naming for account.Manager interface
2023-06-21 16:13:31 +04:00
frank fee033fadb
support mention Non-Latin nickname (#3641)
* support mention non-Latin nickname

* use IndexRune instead of Index

* fix mobile issue #15524

* add another test case
2023-06-21 14:42:14 +08:00
Pascal Precht bf64f97d5a feat: introduce `CheckAllCommunityChannelsPermissions()` API
This API is used to get a permission status of all channels of a given
community.

Clients can use this API to get the provided information for all
community channels with a single RPC call instead of doing one call
for each channel separately.
2023-06-20 10:06:44 +02:00
Pascal Precht 7938297606 feat: introduce `CheckChannelPermissions` API
Similar to `CheckPermissionToJoin()` we now get
a `CheckChannelPermissions()` API.

It will rely on the same `PermissionResponse` types, but gives
information about both `ViewOnlyPermissions` and
`ViewAndPostPermissions`.
2023-06-20 09:42:10 +02:00
Pascal Precht f702b54d08 fix: don't check permissions if revealed accounts aren't legit
This seems to be a bug that was introduced when two features, admin
permissions and "always reveal wallet accounts" where merged.

We need to make sure we **first** check the revealed accounts and only
**then** do we perform permission checks on them. Otherwise we can run
into scenarios where fake addresses are used and users will be accepted
to the community.
2023-06-19 15:27:10 +02:00
Pascal Precht a36202ff30 fix: don't return with error in permissions check when no chain IDs are
found

Turns out that, when we return with an error, instead of
a non-statisfied check permissions response, we can run into cases where
members that should be kicked are not kicked.
2023-06-19 15:26:42 +02:00
Pascal Precht 05e67b1838 fix: add missing private field when converting permission request 2023-06-19 15:26:42 +02:00
frank 0fcf3abd83
Improve/mention input segments (#3631)
* improve calculate input segments relate to mention feature

* bump version

* rename variable
2023-06-19 19:08:45 +08:00
Boris Melnik 1eb92a19a2
feat(communities): Adds muting interval to community (#3549) 2023-06-17 11:19:05 +03:00
frank 9e7f1338f0
refactor function diffText (#3615) 2023-06-16 23:09:37 +08:00
Mikhail Rogachev 51e3d800bb
fix: save revealed accounts regardless of permissions granted (#3609) 2023-06-16 11:10:32 +04:00
Michal Iskierko ccde92377d feat(BurnCollectibles): Add burn collectibles functionality
Change smart contract with new API.
Update gas amount for deployment.
Add Burn() and EstimateBurn() functions.
Add RemainingSupply() functions.

Issue #10816
2023-06-15 15:38:19 +02:00
Andrea Maria Piana 8015cc3e3b Handle race condition when sending envelopes
It happens that an envelope is sent before it's tracked, resulting in
long delays before the envelope is marked as sent.
This commit changes the behavior of the code so that order is now
irrelevant.
2023-06-15 11:28:19 +01:00
Mykhailo Prakhov e058b50a2e
feat: community admin permission functionality (#3569)
* feat: community admin implementation

* fix: code review fixes

* chore: request admin permissions in CheckPermissionToJoin, unit test fix
2023-06-14 10:15:46 -04:00
Pascal Precht 3f231f53e3 Ensure `ChatIds` are handled by token permission requests 2023-06-13 13:42:48 +02:00
mprakhov 83ed3e5ede fix: allow to set token permission as a private 2023-06-13 13:42:48 +02:00
mprakhov 6bc8765d39 chore: new CommunityTokenPermission types 2023-06-13 13:42:48 +02:00
Patryk Osmaczko a402b28b9f chore: resume archives import with delay
closes: status-im/status-desktop#10937
2023-06-13 08:38:15 +02:00
Andrea Maria Piana 37653638bc Make sure the image URL is returned when a replied message is deleted
When we reply to our own message with an image, we didn't set the URL of
the image of the message, which resulted in the image not being
displayed correctly.
2023-06-12 19:22:26 +01:00
Pascal Precht 1a2ca21070 Add chainIds to revealed accounts in memberhsip requests
This commit does a few things:

- Adds a migration that adds chainids to communities_request_to_join_revealed_addresses
- Removes RevealedAddress in favor of RevealedAccount which is now a struct that contains the revealed address, as well as the signature and a list of chain IDs on which to check for user funds
- Changes the logic of sending requests to join a community, such that after creating address signatures, the user node will also check which of the addresses has funds on which networks for the community's token permissions, and add the chainds to the RevealedAccount
- Updates checkPermissionToJoin() such that only relevant chainids are used when checking user's funds. Chain IDs are retrieved from RevealedAccounts and matched against token permission criteria chain IDs
2023-06-12 10:49:29 +02:00
Pascal Precht 511faad0ec fix: don't unsubscribe from community when kicked
There's two scenarios in which we're leaving a community:
We either get kicked or we leave ourselves.

In case of leaving ourselves it's fine to unsubscribe from further
community updates be cause we deliberately chose to leave.

In case of being kicked however, this is different.
Say I'm kicked from a community because its token permissions have
changed, in this case we don't want clients to manually re-subscribe to
the community to get informed when there were further changes.
status-go should rather not unsubscribe if we know for sure we've been
kicked by someone else.
2023-06-12 09:37:12 +02:00
Ibrahem Khalil 47711c4f15
Show mention notification/AC once for a message with multiple images (#3485) 2023-06-10 12:57:40 +04:00
frank b6d4e75cf1
Feat/sync activity center notification (#3535)
fix flaky test: TestRetrieveBlockedContact

resolve conflict when rebase origin/develop

Feat/sync activity center notification (#3581)

* feat: sync activity center notification

* add test

* fix lint issue

* fix failed test

* addressed feedback from sale

* fix failed test

* addressed feedback from ilmotta

go generate ./protocol/migrations/sqlite/...

feat: add updated_at for syncing activity center notification
2023-06-10 10:00:17 +08:00
Sale Djenic 56cad423f0 chore: `forceFetchingBackup` param added to `RequestAllHistoricMessages` and `RequestAllHistoricMessagesWithRetries` functions 2023-06-09 17:42:11 +02:00
Mikhail Rogachev 8589a525a5
Add system message for mutual contact state updates (#3519)
* feat: add mutual state update system message

* feat: send mutual state update on accepting CR

* feat: send mutual state update when removing a contact

* fix: don't send MutualStateUpdateMessage over wire

* fix: mutual state update message text fixed

* fix: new clock to ensure system message after CR and add chat to the response

* feat: add AC notification for contact removal

* feat: replace "sent" mutual state system message with "added"
2023-06-08 16:00:19 +04:00
Stefan 5b6f7226bb feat(wallet) extract json blobs and add custom migration support
Extended the migration process with a generic way of applying custom
migration code on top of the SQL files. The implementation provides
a safer way to run GO code along with the SQL migrations and possibility
of rolling back the changes in case of failure to keep the database
consistent.
This custom GO migration is needed to extract the status from
the JSON blob receipt and store it in transfers table.

Other changes:
- Add NULL DB value tracking to JSONBlob helper
- Index status column on transfers table
- Remove unnecessary panic calls
- Move log_parser to wallet's common package and use to extract token
  identity from the logs

Notes:
- there is already an index on transfers table, sqlite creates one for
  each unique constraint therefore add only status to a new index
- the planned refactoring and improvements to the database have been
  postponed due to time constraints. Got the time to migrate the data
  though, extracting it can be done later for a more efficient
  implementation

Update status-desktop #10746
2023-06-08 13:54:40 +02:00
frank 9e0a12cb32
fix: Mention is not resolved if username or nickname of one is a substring of another (#3571)
* fix mobile issue #16099

* bump version
2023-06-08 14:47:38 +08:00
Patryk Osmaczko 46768e624a feat: add and populate IdentityImage::LocalURL
part of: status-im/status-desktop#10886
2023-06-07 22:32:50 +02:00
Alex Jbanca e8c4b7647f
chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5 (#3559)
* chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5

Changes:
### github.com/mutecomm/go-sqlcipher
1. The improved crypto argorighms from go-sqlcipher v3 are merged in v4
	Tags:
	v4.4.2-status.1 - merge `burn_stack` improvement
	v4.4.2-status.2 - merge `SHA1` improvement
	v4.4.2-status.4- merge 'AES' improvement
2. Fixed `go-sqlcipher` to support v3 database in compatibility mode (`sqlcipher` already supports this) (Tag: v4.4.2-status.3)
3. Upgrade `sqlcipher` to v5.4.5 (Tag: v4.5.4-status.1)

### github.com/status-im/migrate/v4
1. Upgrade `go-sqlcipher` version in `github.com/status-im/migrate/v4`

### status-go
1. Upgrade `go-sqlcipher` and `migrate` modules in status-go
2. Configure the DB connections to open the DB in v3 compatibility mode

* chore(upgradeSQLCipher): Use sqlcipher v3 configuration to encrypt a plain text database

* chore(upgradeSQLCipher): Scanning NULL BLOB value should return nil

Fixing failing tests: TestSyncDeviceSuite/TestPairingSyncDeviceClientAsReceiver;  TestSyncDeviceSuite/TestPairingSyncDeviceClientAsSender

Considering the following configuration:
1. Table with BLOB column has 1 NULL value
2. Query the value
3. Rows.Scan(&dest sql.NullString)

Expected: dest.Valid == false; dest.String == nil
Actual: dest.Valid == true; dest.String == ""

* chore: Bump go-sqlcipher version to include NULL BLOB fix
2023-06-07 08:58:01 +03:00
Sale Djenic 874a0656db fix: profile social links, fix older clock error 2023-06-06 17:05:55 +02:00
Omar Basem 1d4fa29f8e
feat: add images count to reply (#3574)
* feat: add images count to reply
2023-06-06 15:52:07 +04:00
Sale Djenic cff7d2f693 fix: profile social links updated 2023-06-06 09:59:12 +02:00
Patryk Osmaczko 15d2b4fe80 feat: add configurable throttling mechanism for importing msgs
part of: status-im/status-desktop#10815
2023-06-06 08:49:02 +02:00
Pascal Precht ffc1006953 Add permission checks for ENS token permissions
Also add tests for creating and editing token permissions
as well as a test for checking ENS ownership.
2023-06-05 13:10:48 +02:00
Icaro Motta 92b5d831fe
Support unfurling more websites (#3530)
Add support for unfurling a wider range of websites. Most code changes are
related to the implementation of a new Unfurler, an OEmbedUnfurler, which is
necessary to get metadata for Reddit URLs using oEmbed, since Reddit does not
support OpenGraph meta tags. The new unfurler will also be useful for other
websites, like Twitter. Also the user agent was changed, and now more websites
consider status-go reasonably human.

Related to issue https://github.com/status-im/status-mobile/issues/15918

Example hostnames that are now unfurleable: reddit.com, open.spotify.com,
music.youtube.com

Other improvements:

- Better error handling, especially because I wasn't wrapping errors correctly.
  I also removed the unnecessary custom error UnfurlErr.
- I made tests truly deterministic by parameterizing the http.Client instance
  and by customizing its Transport field (except for some failing conditions
  where it's even good to hit the real servers).
2023-06-05 07:46:17 -03:00
Sale Djenic c6192bd26c fix: issues caused by wrong `pairing.proto` fixed 2023-06-05 09:29:34 +02:00
Khushboo-dev-cpp 8b91e3aaaf
chore: Add ColorId for Wallet accounts that can be shared with mobile (#3521) 2023-06-02 17:06:51 +02:00
Boris Melnik 5935339c19
feat(categories): Add muted type to category muting shortcut (#3534) 2023-06-02 14:25:32 +03:00
Jonathan Rainville 3f00869e1f
fix(contacts): add contact to ban and unban responses (#3558) 2023-06-01 13:26:12 -04:00
Boris Melnik af1327f95d
fix(mute): Fix time calculation and skip empty response sending (#3557) 2023-06-01 17:17:42 +03:00
Andrea Maria Piana 7fd9fefdef Add LoginAccount endpoint
This commit adds LoginAccount endpoint.
This makes it consistent with CreateAccount and RestoreAccount as they
use similar config.

The notable difference with the previous endpoint is the API, which is
the same as CreateAccount/RestoreAccount, and the fact that it will
override your networks configuration.

Storing them in the config is now not needed anymore, as that's always
driven from the backend, and we won't allow custom networks in the new
wallet.
2023-06-01 10:05:11 +01:00
Jonathan Rainville b701005f3b
fix(pins): prepare content of pin messages so stickers work (#3523) 2023-05-30 15:05:38 -04:00
Jonathan Rainville b68853c53e
fix(messenger_handler): fix group chat not being set as active (#3545)
Fixes an issue where if a group chat was first received from a non-contact, and later received from a contact, it still wouldn't save it as active.

That's because we checked if we were **newly** added instead of just if we were added. That meant that in the case I described above, the chat would then never have the chance to be set active.
2023-05-30 09:49:46 -04:00
Jonathan Rainville a6285cc827
Fix chats reseting when a Sync signal comes for communities (#3539)
* fix(community): stop re-joining comm when receiving a sync community msg

Fixes an issue with chats being reset. Since joining a community resaves the chats with the synced default value, it resets the sate of the chats, losing the unread messages, the muted state and more.
The solution is to block the re-joining of the community. In the case of the sync, we catch that error and just continue on.

* fix(import): fix HandleImport not saving the chat

Doesn't change much, but it could have caused issues in the future, so since we might have modified the chat, we make sure to save them
Also adds a test

* fix tests
2023-05-29 13:57:05 -04:00
Roman Volosovskyi aded83fc6e
Fix panic on receiving nil message at handleSyncKeypair 2023-05-26 18:43:44 +02:00
Roman Volosovskyi ee4003279a
Changes necessary to restore wallet on mobile app 2023-05-25 21:08:34 +02:00
Sale Djenic 11408f92d7 test: remove flaky test 2023-05-25 19:46:47 +02:00
Sale Djenic 34f5ef031c feat: a profile keypair name follows display name
As part of this commit `UpdateKeypairName` endpoint added,
will be used to rename all but the profile keypairs.
2023-05-25 19:46:47 +02:00
Sale Djenic 20f38bf62b tests: `GetProfileKeypairForTest` from test helper updated and provides higher granularity 2023-05-25 19:46:47 +02:00
Sale Djenic 03f93e9f6c tests: tests affected by accounts improvements are fixed 2023-05-25 19:46:47 +02:00
Sale Djenic eeaaf0ce3f feat: accounts improvements applied
- old `accounts` table is moved/mapped to `keypairs` and `keypairs_accounts`
- `keycards` table has foreign key which refers to `keypairs.key_uid`
- `Keypair` introduced as a new type
- api endpoints updated according to this change
2023-05-25 19:46:47 +02:00
frank 75864528b1
fix: display name/nickname/ens which includes dash is not resolved in the chat when mentioned (#3516) 2023-05-23 18:07:49 +08:00
Mikhail Rogachev 8c90353bc0
fix: create a CR on contact sync with received CR state (#3495)
* fix: create a CR on contact sync with received CR state

* fix: create a CR on contact sync with sent CR state

* Review fixes

* Fix: ignore own contact installation or syncing
2023-05-22 16:22:33 +04:00
Andrea Maria Piana 959e370389 config: add new Push notifications fleet
Also drop no longer existing `eth.test` fleet.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-05-19 17:29:18 +01:00
Andrea Maria Piana 6acb34bab7 Allow setting up wallet with new config
This commit does a few things:

1) Extend create/import account endpoint to get wallet config, some of
   which has been moved to the backend
2) Set up a loop for retrieving balances every 10 minutes, caching the
   balances
3) Return information about which checks are not passing when trying to
   join a token gated community
4) Add tests to the token gated communities
5) Fixes an issue with addresses not matching when checking for
   permissions

The move to the wallet as a background task is not yet complete, I need
to publish a signal, and most likely I will disable it before merging
for now, as it's currently not used by desktop/mobile, but the PR was
  getting to big
2023-05-19 13:23:21 +01:00
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
frank c7aebfeed3
Fix/mobile issue 15899 : error when trying login under updated password (#3493)
* fix: unable to reset password for newly created account using CreateAccountAndLogin

* remove unnecessary print

* add TestCreateAccountAndLogin

* update TestCreateAccountAndLogin

* bump version
2023-05-18 22:00:55 +08:00
Jonathan Rainville 9151aa7f04
fix(unviewed): fix edge case where unviewed count would be -1 (#3491) (#3496) 2023-05-16 12:11:52 -04:00
Jonathan Rainville 34127cd14f
fix(messenger_handler): fix checkForDeletes logic with images (#3477) (#3488) 2023-05-16 11:00:49 -04:00
Ibrahem Khalil 1b6cebac22
Return consistent version of muteTill (#3460) 2023-05-15 18:12:26 +03:00
frank 31d02f5197
fix: Suggested list of mentions does not appear initially when '@' is typed in the editing message composer (#3484)
* fix: Suggested list of mentions does not appear initially when '@' is typed in the editing message composer

* bump version
2023-05-15 20:15:10 +08:00
Mikhail Rogachev d55adaf3a4
fix: marshal dismissed contact request state properly (#3482) 2023-05-15 11:29:33 +04:00
frank 39f486a47a
optimize mention (#3479)
* optimize mention

* bump version
2023-05-15 15:21:41 +08:00
frank 0197e6c484
Feat/sync local deleted message (#3476)
* sync local deleted messages

* rebase

* add REPLACE

* fix lint

* defer rows.Close() / rename function

* add local pair test

* replace unused clock with _
2023-05-12 16:31:34 +08:00
Patryk Osmaczko fd05016c6e fix: mark imported messages as seen
fixes: status-im/status-desktop#10600
2023-05-10 18:51:48 +02:00
Ibrahem Khalil ff75280b30
Return compressed key with emojis (#3410) 2023-05-10 13:45:42 +03:00
Mikhail Rogachev 40e57f1127
Test: add TestBobSendsContactRequestAfterDecliningOneFromAlice test case (#3468) 2023-05-10 12:01:26 +04:00
Sale Djenic 189c7a6919 chore: `keypairs` package which actually refers to keycards is renamed to `keycards` package
This commit renames few api endpoints:
- old `AddMigratedKeyPairOrAddAccountsIfKeyPairIsAdded` renamed to `AddKeycardOrAddAccountsIfKeycardIsAdded`
- old `GetAllMigratedKeyPairs` renamed to `GetAllKnownKeycardsGroupedByKeyUID`
- old `GetMigratedKeyPairByKeyUID` renamed to `GetKeycardByKeyUID`
- old `DeleteKeypair` renamed to `DeleteAllKeycardsWithKeyUID`
2023-05-10 09:27:51 +02:00
frank 98d3b4198b
sync message for `delete for me` should not be sent to someone else (#3462)
* sync message for `delete for me` should not be sent to someone else

* addressed feedback from review

* remove LocalChatID

* bump version
2023-05-09 20:54:56 +08:00
Sale Djenic 31144ed5a3 chore: `password` removed from the `DeleteAccount` function
Asking for a password when removing a key is not necessary.
2023-05-09 10:48:44 +02:00
Volodymyr Kozieiev a669e7d038
More fields added to ChatPreview. LastMessage now can be a deleted message (#3397) 2023-05-08 18:02:54 +01:00
Patryk Osmaczko 8a95dab5eb fix: covert curated communities pubkeys properly 2023-05-08 10:58:36 +02:00
Jonathan Rainville edf8869ff2
fix(communities/manager): use sync.map to fix concurrent map writes err (#3445) 2023-05-05 14:13:55 -04:00
Jonathan Rainville b7767ea63c
fix(communities/manager): make historyArchiveTasks a SyncMap (#3448)
Fixes #3447
2023-05-05 12:40:18 -04:00
Michal Iskierko 4935848287 fix(GetAllCommunityTokens): Add GetAllCommunityTokens function
Issue #10254
2023-05-05 18:14:55 +02:00
osmaczko 83998272dd
feat: add featured communities (#3459) 2023-05-05 11:55:32 -04:00
Andrea Maria Piana fe75054218 Query only once mailservers per cycle 2023-05-05 16:49:06 +01:00
mprakhov 57b2432290 Community encryption should automatically be enabled/disabled depending on community permissions (closed/open) 2023-05-05 17:32:39 +02:00
Igor Sirotin 44ded4dd64
fix(Messenger): Count SystemMessagePinnedMessage as unviewed (#3449) 2023-05-05 18:26:19 +03:00
Andrea Maria Piana 46bac20260 Fix installation test 2023-05-05 12:41:28 +01:00
frank a8fe8edd45
Fix/flaky test sync installation (#3452)
* fix flaky test: sync installation

* bump version

* fix error description
2023-05-05 14:44:26 +08:00
Boris Melnik ebf997bf63
fix(communities): Ignore requests to join from banned user (#3030) 2023-05-03 18:01:55 +03:00
Richard Ramos 975883407f refactor: improve mailserver batch handling 2023-05-02 07:27:57 -04:00
Richard Ramos 2ec97ee861 feat: use 24h batches for retrieving message history 2023-05-02 07:25:59 -04:00
Patryk Osmaczko dd6cb5394d chore: add index idx_user_messages_unseen 2023-05-02 09:21:58 +02:00
Richard Ramos 323a6ce6dd test: processMailserverBatch 2023-04-30 10:04:04 -04:00