Commit Graph

102 Commits

Author SHA1 Message Date
Boris Melnik 3e7d1a5f34
fix(migration): Fixes mute_till migration (#3653) 2023-06-22 18:49:58 +03:00
Ibrahem Khalil d3e650d5e5
Make chat muting only works on one chat and not to completely block PNs from a user (#3647) 2023-06-22 08:06:32 +03:00
Boris Melnik 1eb92a19a2
feat(communities): Adds muting interval to community (#3549) 2023-06-17 11:19:05 +03:00
Mykhailo Prakhov e058b50a2e
feat: community admin permission functionality (#3569)
* feat: community admin implementation

* fix: code review fixes

* chore: request admin permissions in CheckPermissionToJoin, unit test fix
2023-06-14 10:15:46 -04:00
Roman Volosovskyi ee4003279a
Changes necessary to restore wallet on mobile app 2023-05-25 21:08:34 +02:00
Andrea Maria Piana 6acb34bab7 Allow setting up wallet with new config
This commit does a few things:

1) Extend create/import account endpoint to get wallet config, some of
   which has been moved to the backend
2) Set up a loop for retrieving balances every 10 minutes, caching the
   balances
3) Return information about which checks are not passing when trying to
   join a token gated community
4) Add tests to the token gated communities
5) Fixes an issue with addresses not matching when checking for
   permissions

The move to the wallet as a background task is not yet complete, I need
to publish a signal, and most likely I will disable it before merging
for now, as it's currently not used by desktop/mobile, but the PR was
  getting to big
2023-05-19 13:23:21 +01:00
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
frank a2ed1b78dd
feat:support backup/sync ens name (#3415) 2023-04-26 23:37:18 +08:00
Andrea Maria Piana 7650f3003e Fix some issues with pinned messages
There were a couple of issues on how we handle pinned messages:

1) Clock of the message was only checked when saving, meaning that the
   client would receive potentially updates that were not to be
   processed.
2) We relied on the client to generate a notification for a pinned
   message by sending a normal message through the wire. This PR changes
   the behavior so that the notification is generated locally, either on
   response to a network event or client event.
3) When deleting a message, we pull all the replies/pinned notifications
   and send them over to the client so they know that those messages
   needs updating.
2023-04-25 16:02:48 +01:00
frank 272173c939
feat:support backup/sync social settings (#3407)
* feat:support backup/sync social settings

* update binddata.go

* fix lint issue

* addressed feedback from @saledjenic @osmaczko

* addressed feedback from @saledjenic @osmaczko

* add corresponding test for local pairing

* addressed feedback from @saledjenic
2023-04-20 06:59:09 +08:00
Samuel Hawksby-Robinson b47bc730a0 reset superfluous go mod tidy changes 2023-04-17 12:01:38 +01:00
Samuel Hawksby-Robinson 17bc1e9f5d Rough initial proof of concept for udp network checking 2023-04-17 12:01:38 +01:00
Ibrahem Khalil 719af90fcc
Add muted_till param for chats (#3258) 2023-04-16 17:06:00 +02:00
frank 5fd9e93e9c
backup/sync if seed phrase removed (#3385) (#3384)
backup/sync bio
2023-04-13 08:45:25 +08:00
Ibrahem Khalil d0766e3023
Update album_images_count column on receiver side (#3375) 2023-04-05 15:24:55 +02:00
Boris Melnik 696e061861
fix(unread_count): Skip extra count of new messages for album of images (#3345)
* fix(unread_count): Skip extra count of new messages for album of images

* fix(unread_count): Update migration files
2023-03-31 12:15:06 +03:00
Ibrahem Khalil bd593373b2
Add album count key to messages (#3347) 2023-03-30 12:02:20 +02:00
Andrea Maria Piana 44a0f5b74d
Add collapsed community categories 2023-03-14 17:13:21 +00:00
Icaro Motta 999d8c0ee0
Support soft deletion for activity center notifications (#3201)
Adds a new column named `deleted` to the table `activity_center_notifications`. 

Related PR in Mobile https://github.com/status-im/status-mobile/pull/15106 for a lot more details of the feature.

Why? Relying on the `dismissed` column for soft deletion is no longer viable because the mobile & desktop clients should display dismissed notifications (sometimes), hence the need for a new column to truly represent soft deletion.
2023-02-24 20:47:04 -03:00
Andrea Maria Piana 11db9f2119
Add test for everyone tag & fix migration order
A migration was added out-of-order, which meant that in clients who
had already run the migration after, it would be skip.
This commit re-adds the migration so it's run, tested against an empty
account and one that had already migrated.
2023-02-24 10:18:26 +00:00
Mikhail Rogachev f25088483c
feat: Add seen/unseen activity center setting (#3148)
* feat: Add seen/unseen activity center state

* feat: ActivityCenterState for grouping ActivityCenter unread messages cnt and seen state

* feat: always use messenger's addActivityCenterNotification & add state to the response

* Remove unused activity center endpoints form api and fix test
2023-02-17 14:08:08 +04:00
Andrea Maria Piana cefa0089dc
Split clock values for contacts 2023-01-31 15:43:00 +00:00
Stefan 5ff029dd57 feat: add balance_history DB table and update migration scripts
Updates status-desktop #8226
2023-01-25 22:25:50 +04:00
Igor Sirotin f9faac4293
feat: ens_usernames database (#3066) 2023-01-20 13:34:30 +03:00
Parvesh Monu d40290a649
Persist switcher cards (#3049) 2023-01-05 21:41:57 +05:30
Andrea Maria Piana c85df62d3c Sync community encryption keys 2022-11-15 10:11:01 +00:00
Boris Melnik 63d46481fd
feat(communities): Cancel request to join community (#2915) 2022-10-28 11:41:20 +03:00
Pascal Precht 0bdb596d3b feat: introduce discord import tool 2022-10-28 09:52:26 +02:00
Andrea Maria Piana a89f4b2d71 Add read/unread messenger functions for activity center 2022-10-26 17:19:44 +01:00
Mikhail Rogachev 60cb15739d
feat(ActivityCenter): Add community membership AC notifications (#2886)
* feat(ActivityCenter): Add community request AC notification

* feat(ActivityCenter): Add CommunityID to AC notification

* feat(ActivityCenter): Add membership status for community membership AC notifications

* feat(ActivityCenter): Add tests for community notifications and fix naming

* Add notification for kicked from community action

* feat(ActivityCenter): Fix for missing notification objects for tests
2022-10-26 02:06:20 +04:00
Andrea Maria Piana bef211b5d6 Send all encryption keys 2022-10-20 12:19:44 +01:00
Stefan b9d6e6d8c8 feat: sync saved addresses
Main changes:
- Extend saved addresses DB with sync info: sync timestamp, update timestamp
and soft removed flag
- Create custom protobuf message payload to sync saved addresses
- Cleanup saved addresses on each start of messenger, by deleting
soft removed older entries
- Sync all saved addresses on Messenger.SyncDevices calls
- Sync particular changes to saved addresses
- Add SavedAddressManager instance to messenger
  - Note, can't find a clean way to pass the SavedAddressManager to the
  messenger, so we create another one
- Add tests for sync and new DB API

Closes: #7229
2022-10-10 11:26:35 +02:00
Stefan daee116878 fix: compiling chat_messsage.proto due to unscoped ContentType 2022-10-07 12:49:04 +02:00
Pascal Precht 9ee2967e78 feat(message_persistence): add discord message author image payload fields
This also adds some persistence APIs to operate on the new field (later
needed for the discord imoprt tool)
2022-09-19 13:47:16 +02:00
Stefan 5b0d8a7c4e fix: workaround for wrong renaming of migration file for keypairs table
Restore the old, previously renamed, 1662447680_add_keypairs_table.up.sql
file while keeping the current one for those who already migrated to the
new one. The extra migration is noop and saves to keep consistency in
the user data states history.
2022-09-16 12:52:01 +02:00
Stefan 9db69df9a0 feat: updated migrations for merging favourites with saved addresses
Required by #6546
2022-09-12 13:46:04 +02:00
Patryk Osmaczko 65be6f2b96 feat: add and distribute `chatIdentity.FirstMessageTimestamp`
`FirstMessageTimestamp` enables members of the community to determine if
there are any messages they can fetch on the community channel(chat).

`FirstMessageTimestamp` is advertised by admin for each community chat
through `CommunityDescription`. It assumes admin is online frequently
enough to capture the first channel message.

For existing communities admin determines first message timestamp by
finding oldest chat message in its local database.

task: status-im/status-desktop#6731
2022-09-09 08:59:39 +02:00
Michal Iskierko c277e8addc fix: change migration timestamp of group chat add image feature 2022-09-01 17:55:46 +02:00
Michal Iskierko e3f451cd79 feat: Handle image in group chat
Add image_payload column to chats table.
Add Base64Image to chat struct.
Add ImageChange event to propagate image.
Change EditChat API - use CroppedImage.
Process and crop image in EditGroupChat.
2022-09-01 12:19:01 +02:00
Roman Volosovskyi f43f43cc30
[#13672] Sync contact request decision 2022-08-24 16:08:41 +02:00
Samuel Hawksby-Robinson 38c8e8a1fe Added payload marshal and unmarshal to/from protobuf 2022-08-22 13:27:59 +01:00
siddarthkay 924820c14a Implement wallet connect session CRUD API 2022-08-19 12:32:00 +01:00
Pascal Precht d0e0deac95 feat: introduce `discord_message_authors` persistence APIs
This introduces a new table to store discord message authors.
The main reason this table is being introduce is so that we don't have
to duplicate discord message author information in the `user_messages`
table when importing discord communities (ongoing work).

In addition to the table there are also two new APIs on the messenger
persistence layer (which are later used in the import logic):

- `HasDiscordMessageAuthor`
- `SaveDiscordMessageAuthor`

Closes #2759
2022-08-10 10:13:55 +02:00
Michal Iskierko 31671ea040 feat: distribute color info when creating group chat or changing chat color
Regenerating protocol protobuf files - membership_update_message.proto change.

Issue #5982
2022-08-08 10:33:38 +02:00
Pascal Precht 9c568c58cf feat: introduce messenger APIs to extract discord channels
As part of the new Discord <-> Status Community Import functionality,
we're adding an API that extracts all discord categories and channels
from a previously exported discord export file.

These APIs can be used in clients to show the user what categories and
channels will be imported later on.

There are two APIs:

1. `Messenger.ExtractDiscordCategoriesAndChannels(filesToimport
   []string) (*MessengerResponse, map[string]*discord.ImportError)`

   This takes a list of exported discord export (JSON) files (typically one per
   channel), reads them, and extracts the categories and channels into
   dedicated data structures (`[]DiscordChannel` and `[]DiscordCategory`)

   It also returns the oldest message timestamp found in all extracted
   channels.

   The API is synchronous and returns the extracted data as
   a `*MessengerResponse`. This allows to make the API available
   status-go's RPC interface.

   The error case is a `map[string]*discord.ImportError` where each key
   is a file path of a JSON file that we tried to extract data from, and
   the value a `discord.ImportError` which holds an error message and an
   error code, allowing for distinguishing between "critical" errors and
   "non-critical" errors.

2. `Messenger.RequestExtractDiscordCategoriesAndChannels(filesToImport
   []string)`

   This is the asynchronous counterpart to
   `ExtractDiscordCategoriesAndChannels`. The reason this API has been
   added is because discord servers can have a lot of message and
   channel data, which causes `ExtractDiscordCategoriesAndChannels` to
   block the thread for too long, making apps potentially feel like they
   are stuck.

   This API runs inside a go routine, eventually calls
   `ExtractDiscordCategoriesAndChannels`, and then emits a newly
   introduced `DiscordCategoriesAndChannelsExtractedSignal` that clients
   can react to.

   Failure of extraction has to be determined by the
   `discord.ImportErrors` emitted by the signal.

**A note about exported discord history files**

We expect users to export their discord histories via the
[DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter/wiki/GUI%2C-CLI-and-Formats-explained#exportguild)
tool. The tool allows to export the data in different formats, such as
JSON, HTML and CSV.

We expect users to have their data exported as JSON.

Closes: https://github.com/status-im/status-desktop/issues/6690
2022-08-04 14:34:23 +02:00
Pascal Precht f6c9ec7838 refactor: Require request access for all communities
This commit introduces a few changes regarding users accessing
communities:

While the APIs still exist, community invites should no longer be
used, instead communities should merely be "shared".
Sharing a community to users allows users to "join" the community,
which in reality makes them request access to that community.
This means, users have to request access to any community, even if
the community has permissions set to NO_MEMBERSHIP
Only difference between ON_REQUEST and NO_MEMBERSHIP is that
ON_REQUEST communities require manual approval of the owner/admin
to access a community. NO_MEMBERSHIP communities accept
automatically (as soon as owner/admin receives the request).
This also implies that users are no longer optimistically added to the
member list of communities, but only after they have been accepted.

This introduces a bit of a message ping-pong for users to know that
someone is now part of a community
2022-07-08 13:16:12 +02:00
Vitaliy Vlasov 3dee94e505 Wallet sync for generated accounts 2022-07-06 19:24:43 +03:00
Richard Ramos 0322ac497b
feat: contact verification request (#2586)
fix: add verification request to response

fix: code review

add missing functions and simplify timestamp usage

fix: sync verification requests

feat: add endpoint to fetch all received verification requests

feat: add signal when trusting verification request

Co-authored-by: Jonathan Rainville <rainville.jonathan@gmail.com>
2022-07-05 15:49:44 -04:00
Vitaliy Vlasov 4f722b6fe8 Communities encryption integration 2022-06-17 21:24:39 +03:00
Pascal Precht f3af6aa4dc feat: introduce `AddBookmark` and `RemoveBookmark` APIs to messenger
These APIs are being introduced to address #2706 and #2704, provided
that clients will move to using these APIs instead of the currently
provided equivalent APIs in the browser service.

The `bookmarks` table is being extended with a `deleted_at` field which
can later be used for garbage collection, as "removing" a bookmark is
merely soft deletion and doesn't actually remove the data from the
database.

In addition to those APIs adding and soft deleting bookmark entries,
they also automatically perform a sync operation to ensure that
bookmarks are synced in real-time (#2704).

Closes #2706, #2704
2022-06-17 12:29:03 +02:00