Commit Graph

78 Commits

Author SHA1 Message Date
Pascal Precht b4bdfd3df6 Keep track of last seen magnetlink
This is to prevent processing magnetlinks if they haven't changed
2022-12-20 15:16:08 +01:00
Pascal Precht f5dfa58602 refactor(communities): better archive download handling
This commit makes a few changes to the community history archive
download routine to make it more robust:

1. Prior to this commit, even when there were no archives to be
   downloaded, we were still trying to extract messages from archive
   data.
2. Logs have been improved as they were sometimes showing confusing
   information
3. We now handle interruption of ongoing download + data import much
   better in case of multiple magnetlinks being processed in roughly the
   same time.
4. We now keep track of which archive has been successfully imported
   into the database. Without this, Status would consider any downloaded
   archives as "done" even though they haven't actually been imported
   into the database yet. This way Status should be able to pick up its
   work were it left of the last time, in case a user closes the app, or
   another magnetlink interrupts the ongoing process.
2022-12-13 16:13:15 +01:00
Pascal Precht 4bddfbd466 chore: better community archive protocol logs 2022-12-13 16:13:15 +01:00
Pascal Precht 8ac154ee0c fix: don't keep Messenger from starting when torrent client fails to
start

There are various cases that could make the torrent client fail to
start. We don't want this to keep the entire messenger from starting.
2022-12-06 18:19:36 +01:00
Michal Iskierko 8fc7d921a7 fix(ImportCommunity): Setting admin roles during import community.
Issue 
2022-12-06 10:25:21 +01:00
Pascal Precht 22b5690cda feat: add signal indicating that history messages are being imported
In order to give clients more insights about archive messages being
processed, we're adding this additional signal that informs clients when
the import of downloaded history archive messages has started.
2022-12-05 10:51:53 +01:00
frank 684e9654de
Allow owner/admin to delete messages of a community () 2022-12-02 19:34:02 +08:00
Pascal Precht 194f26f3b4 feat: add signal for starting history archive downloads 2022-12-01 16:12:00 +01:00
Boris Melnik 63d46481fd
feat(communities): Cancel request to join community () 2022-10-28 11:41:20 +03:00
Pascal Precht 0bdb596d3b feat: introduce discord import tool 2022-10-28 09:52:26 +02:00
Mikhail Rogachev 60cb15739d
feat(ActivityCenter): Add community membership AC notifications ()
* feat(ActivityCenter): Add community request AC notification

* feat(ActivityCenter): Add CommunityID to AC notification

* feat(ActivityCenter): Add membership status for community membership AC notifications

* feat(ActivityCenter): Add tests for community notifications and fix naming

* Add notification for kicked from community action

* feat(ActivityCenter): Fix for missing notification objects for tests
2022-10-26 02:06:20 +04:00
Pascal Precht 0ccdec2985 feat(CommunitiesManager): introduce `CreateHistoryArchiveTorrentFromMessages` API
Prior to this commit we had a `CreateHistoryArchiveTorrent()` API which
takes a `startDate`, an `endDate` and a `partition` to create a bunch of
message archives, given a certain time range.

The function expects the messages to live in the database, which means,
all messages that need to be archived have to be saved there at some
point.

This turns out to be an issue when importing communities from third
party services, where, sometimes, there are several thousands of messages
including attachment payloads, that have to be save to the database
first.

There are only two options to get the messages into the database:

1. Make one write operation with all messages - this slow, takes a long
   time and blocks the database until done
2. Create message chunks and perform multiple write operations - this is
   also slow, takes long but makes the database a bit more responsive as
   it's many smaller operations instead of one big one

Option 2) turned out to not be super feasible either as sometimes,
inserting even a single such message can take up to 10 seconds
(depending on payload)

Which brings me to the third option.

**A third option** is to not store those imported messages as waku
message into the database, just to later query them again to create the
archives, but instead create the archives right away from all the
messages that have been loaded into memory.

This is significantly faster and doesn't block the database.

To make this possible, this commit introduces
a `CreateHistoryArchiveTorrentFromMessages()` API, and
a `CreateHistoryArchiveTorrentFromDB()` API which can be used for
different use cases.
2022-10-24 13:28:55 +02:00
Pascal Precht 7eb66d09e7 feat(communities): add encryption suppord to archive protocol 2022-10-24 12:35:38 +02:00
Pascal Precht b2dce92f3f fix(Communities): correct query params order when updating community
settings

Turns out `UpdateCommunitySettings()` has never worked. Two parameters
where in the wrong order, cause the SQL statement to never find the row
it has to update.
2022-10-24 12:02:52 +02:00
Pascal Precht 69e84b5673 feat(communities): retry downloading archive data
When fetching torrent info after receiving a magnet link,
it can happen that the request times out.

We want to retry downloading the data again at least once more
before giving up
2022-10-17 11:33:02 +02:00
Pascal Precht dd49c4c954 feat(communities): add archiveLogger for stdout logs
The default logger writes to `geth.log`, which makes debugging
the archive protocol pretty hard.

This adds an additional logger that logs to stdout, while keeping
the default logger intact for production.
2022-10-14 16:31:21 +02:00
Richard Ramos d4d35710be fix: community encryption was not being marshalled 2022-10-10 13:11:10 +02:00
Patryk Osmaczko 73a45dd58e feat: add `SpectateCommunity` api
- added `SpectateCommunity` endpoint, it is supposed to be used in
  scenarios where we want to "Go to public Community" and see its
  content without joining
- added `spectated` field to `Community`, it means we are observing the
  community and its chats but we are not members

Use case:
https://github.com/status-im/status-desktop/issues/7072#issuecomment-1246560885
2022-10-06 21:21:37 +02:00
Pascal Precht ffc9aea24f fix: ensure torrent download does not finish too early 2022-10-05 13:13:16 +02:00
Pascal Precht c01ad05525 feat(common): add `ThirdPartyID` to `Message` and `protobuf.WakuMessage`
Usually, message IDs are generated by their payload and signature and
in receiving nodes calculated in based on the same data as well.

There's no ID attached to messages in-flight.

This turns out to be a bit of a problem for messages that are being
imported from third party systems like discord, as the conversion
and saving of such messages and handling of their possible assets and
attachments are done in separate steps, which changes the message
payloads after their IDs have been generated.

Hence, we're introducing a `ThirdPartyID` property to `common.Message`
and `protobuf.WakuMessage` so receiving nodes of such messages (via the
archive protocol primarily) can easily detect third party/imported
messages and give them special treatment.
2022-09-29 11:15:47 +02:00
Pascal Precht 81e279a055 chore(community): community itself has admin rights
This might look like a weird requirement at a fist glance.
The reason this is needed, is because some message signals require
admin rights to take effect (e.g. PinMessage).

When messages are imported from third-party services,
translated to status messages, signed by the community, and eventually distributed
via the archive protocol, we need to ensure that messages signed
by the community itself are considered as admin privileges as well,
so they can be correctly replayed into the database.
2022-09-28 13:03:56 +02:00
Richard Ramos bf9abfc350
feat: make kdf iterations configurable when creating a multiaccount () 2022-09-27 16:27:20 -04:00
Patryk Osmaczko e4ca8a256c fix: remove ourselves from members when leaving community
Otherwise clients see invalid members count in invitation bubbles.
2022-09-21 08:35:52 +02:00
Pascal Precht eb4ab9316c fix(communities): ensure existing archives are seeded on startup 2022-09-19 15:34:29 +02:00
Pascal Precht fd4488e504 feat: add signal to inform clients when all archives have been downloaded and handled
This adds a new `DownloadingHistoryArchivesFinished` signal to the
family of community archive signals. It's emitted when all to be
downloaded archives have been downloaded and handled.
2022-09-16 09:43:49 +02:00
Patryk Osmaczko 65be6f2b96 feat: add and distribute `chatIdentity.FirstMessageTimestamp`
`FirstMessageTimestamp` enables members of the community to determine if
there are any messages they can fetch on the community channel(chat).

`FirstMessageTimestamp` is advertised by admin for each community chat
through `CommunityDescription`. It assumes admin is online frequently
enough to capture the first channel message.

For existing communities admin determines first message timestamp by
finding oldest chat message in its local database.

task: 
2022-09-09 08:59:39 +02:00
Patryk Osmaczko 7e25a6125f feat: introduce and distribute RequestToLeave community
closes: 
2022-08-26 11:25:33 +02:00
Pascal Precht 9b04633bb7 feat(communities): introduce `publish` flag in community manager APIs
This is so that we can control whether we want to publish the community
when it, or it's categories and channels, are created.

This is needed for the discord import so that we can create communities,
channels and categories without publishing the community and have it
show up in UIs too early.
2022-08-25 12:59:30 +02:00
Michal Iskierko e6a3f63ec7 feat: Add DeclinedRequestsToJoinForCommunity to API
Issue 
2022-08-22 13:43:26 +02:00
Patryk Osmaczko e5e07408e2 feat: share image/username on separate topic on a community
closes: 
2022-07-20 10:35:19 +02:00
Pascal Precht f6c9ec7838 refactor: Require request access for all communities
This commit introduces a few changes regarding users accessing
communities:

While the APIs still exist, community invites should no longer be
used, instead communities should merely be "shared".
Sharing a community to users allows users to "join" the community,
which in reality makes them request access to that community.
This means, users have to request access to any community, even if
the community has permissions set to NO_MEMBERSHIP
Only difference between ON_REQUEST and NO_MEMBERSHIP is that
ON_REQUEST communities require manual approval of the owner/admin
to access a community. NO_MEMBERSHIP communities accept
automatically (as soon as owner/admin receives the request).
This also implies that users are no longer optimistically added to the
member list of communities, but only after they have been accepted.

This introduces a bit of a message ping-pong for users to know that
someone is now part of a community
2022-07-08 13:16:12 +02:00
Boris Melnik 3e65e36fa6
feat: Add mute category methods () 2022-07-06 19:16:19 +03:00
Mikhail Rogachev 7ad0057003
feat(@communities): Add saving tags on community edit () 2022-07-04 16:56:00 +03:00
Michal Iskierko 72d2a97449 feat: unban user in community
Issue 
2022-07-04 09:02:33 +02:00
Richard Ramos 2873e65a61
feat: community tags ()
Signed-off-by: Richard Ramos <info@richardramos.me>

Co-authored-by: MishkaRogachev <mishkarogachev@gmail.com>
2022-06-24 09:40:12 -04:00
Boris Melnik 8063152657
fix(community-edit): Save permissions when community edited ()
Part of: https://github.com/status-im/status-desktop/issues/6073
2022-06-21 13:43:49 -04:00
Vitaliy Vlasov 4f722b6fe8 Communities encryption integration 2022-06-17 21:24:39 +03:00
Pascal Precht 9d5bcc3c60 feat: sync `CommunitySettings` across paired devices
This commit introduces a new `clock` field in the
`communities_settings` table so that it can be leveraged for syncing
community settings across devices.

It alsoe exends existing `syncCommunity` APIs to generate
`SyncCommunitySettings` as well, avoiding sending additional sync messages
for community settings.

When editing communities however, we still sync community settings
explicitly are we aren't syncing the community itself in that case.
2022-06-16 12:45:05 +02:00
Richard Ramos 9430f494be
fix: handle nil descriptions in communities for chat_getChats () 2022-06-08 08:46:52 -04:00
Richard Ramos c3b0582cc9
feat: fetch curated communities from smart contract on optimism () 2022-06-02 08:17:52 -04:00
Patryk Osmaczko ee7025a573 feat: add intro/outro message to communities 2022-05-31 11:41:26 +02:00
Stefan 63e58ba035 feat: add banner support for communities
Add banner image as a special `IdentityImage` beside "thumbnail" and "large"

Banner input cropped image processing

- Resize to keep in the limits of `BannerDim`
- Encode to match the file size limits define for banner
- Don't scale up. This can be done efficiently in the UI

Changes to `images` module

- Refactor `EncodeToBestSize` as `EncodeToLimits` to accept arbitrary dimensions
  and allow for custom size
- Define `DimensionLimits` for banner not to exceed 450 KB and a rough estimate
  for the ideal size
2022-05-27 14:42:14 +03:00
Pascal Precht 714c03c635 feat: introduce `CommunityAdminSettings` in `CommunityDescription`
This allows to store community admin settings that are meant to be propagated
to community members (as opposed to the already existing
`CommunitySettings` which are considered local to every account).

The first setting introduced as part of this commit is one that enables
community admins to configure whether or not members of the community
are allowed to pin messages in community channels.

Prior to this commit, this was not restricted at all on the protocol
level and only enforced by clients via UI (e.g. members don't see an
option to pin messages, although they could).

This config setting now ensures that:

1. If turned off, members cannot send a pin message
2. If turned off, pin messages from members are not handled/processed

This is needed by https://github.com/status-im/status-desktop/issues/5662
2022-05-18 09:58:11 +02:00
Pascal Precht 43d6305157 Handle history archive magnetlink messages
This introduces the ability for status notes to handle community
history archive magnetlinks. To make this work, a few things are needed:

1. A new database table has been introduced to store message archive
   hashes. This is necessary so status nodes can determine whether or
   not they need to download a certain archive
2. The messenger's `handleRetrievedMessages()` has been exteded to take
   magnetlink messages into account
3. New APIs were added to download torrent data given a magnetlink and
   also to extract messages from downloaded archives, which are then
   later fed to `handleRetrievedMessages`

Closes 
2022-05-10 11:27:31 +02:00
Roman Volosovskyi 9de4eb3c8a
Backup deleted communities 2022-04-14 18:01:07 +02:00
Pascal Precht 7032fc9dcc Introduce community history archive routine
This introduces logic needed to:

- Create WakuMessageArchives and and indices from store waku messages
- History archive torrent data to disk and create .torrent file from
  that
- Seed and unseed history archive torrents as necessary
- Starting/stopping the torrent client
- Enabling/disabling community history support for individual components
  and starting/stopping the routine intervals accordingly

This does not yet handle magnet links ()

Closes 
2022-04-06 13:00:33 +02:00
Pascal Precht 364838532f Add torrent library dependency 2022-04-06 11:48:16 +02:00
Andrea Maria Piana 8a5f77dc37 Address linter issues 2022-03-28 13:14:12 +01:00
Pascal Precht 3b3921f483 Make community owners store waku message payloads
This is needed so that when they are bundled into archives, receiving
nodes can still verify the messages payload using its signature.

This commit introduces a new `waku_messages` table and APIs to store
such messages. Waku message payload is store for any message that has
a topic that matches any of the admin communities chats.

Closes 
2022-03-28 11:24:23 +02:00
Pascal Precht bf8e71cfa9 Introduce `CommunitySettings` to store community related settings
These are used to store settings for individual communities a
user is part of, either as member or as owner.
This included whether or not the community history archive protocol
is enabled.

This adds a new `CommunitySettings` type and adds
a migration script that introduces a new `communities_settings`
table.

It also extends the `MessengerResponse` type to include
`CommunitySettings` which are honored when communities are being
added, edited, joined or left.

Lastly, this adds a new RPC API to retreive the settings.

Closes 
2022-03-22 12:33:40 +01:00