This introduces an addition constraint to archive generation, in which the payload + signature size of all partitioned message that go into an archive should not exceed a certain
threshold.
This is to ensure that archives won't get too big when they are later read into memory.
Summary
=======
- [x] Changes endpoint ActivityCenterNotificationsBy to support fetching
multiple types of notification in a single query.
- [x] Adds endpoint UnreadAndAcceptedActivityCenterNotificationsCount to
allow the mobile client to fetch the count of unread & accepted
notifications.
- [x] Add `golangci-lint` to Nix shell. This was possible since PR
https://github.com/status-im/status-go/pull/3087 was merged.
Notes
=====
- If you'd like to understand why these changes are needed, please see
the mobile PR https://github.com/status-im/status-mobile/pull/14785,
or issue https://github.com/status-im/status-mobile/issues/14712
- All changes should be completely backwards compatible, and there
should be no impact for the desktop app.
- The mobile client has been already tested using this branch.
This adds the functionality that history archives continue to be imported
in case the import has been interrupted the last time the app/client
was running.
This typically happens when users don't wait for an ongoing import to finish,
which sometimes can take a while. Users then close the app/kill the client
which leaves the database in a state where there's downloaded archives that
haven't been fully imported.
Prior to this change, the node will have to wait until it receives a new
magnetlink that it hasn't seen before, until it processes imports again.
This can take several days.
Now, it will check on startup if there are any archives left to be imported
and resumes the import from there.
Instead of loading the entire torrent file into memory when trying
to extrract active messages, we now only read the chunks that are
necessary to decode any individual archive and then process
extracted messages in chunks.
This doesn't introduce a max cap of allowed memory yet, since the
chunk size depends entirely on the size of the archive, but this
will be done soon.
Because `QuotedMessage` doesn't include imported message data,
some of the author information in imported messages is lost in
frontends.
This commit adds a `discordMessage` (soon replaced by `importedMessage`)
to `Quotedmessage`, although only hydrated with a subset of data,
namely author display name and avatar URL, as those are the only ones
needed by front-end atm.
* feat(ActivityCenter): Add missing AC notifications for verification requests
* fix(Contacts): Fix updates for trusted and untrustworthy statuses
* feat(ActivityCenter): Add test for trusted verification request and AC notifications
* feat(Contacts): Trusted and untrustworthy statuses should be unknown for verificated side
Changes applied here introduce backing up profile data (display name and identity
images) to waku and fetch them from waku. Information about those data is sent
as a separate signal to a client via `sync.from.waku.profile` signal.
New signal `sync.from.waku.progress` is introduced which will be used to notify a client
about the progress of fetching data from waku.
When discovery fails to be seeded with bootstrap/fallback nodes, it
never recovers.
This commit changes the behavior so that status-go retries fetching
bootnodes, and restarts discovery when that happens.
This is done so that when member join a community by being accepted by
the community owner, they also receive the most up-to-date magnetlink
along with it.
This commit makes a few changes to the community history archive
download routine to make it more robust:
1. Prior to this commit, even when there were no archives to be
downloaded, we were still trying to extract messages from archive
data.
2. Logs have been improved as they were sometimes showing confusing
information
3. We now handle interruption of ongoing download + data import much
better in case of multiple magnetlinks being processed in roughly the
same time.
4. We now keep track of which archive has been successfully imported
into the database. Without this, Status would consider any downloaded
archives as "done" even though they haven't actually been imported
into the database yet. This way Status should be able to pick up its
work were it left of the last time, in case a user closes the app, or
another magnetlink interrupts the ongoing process.
This creates a smoother experience for users when they leave a community since they can see the exact same messages they had before without having to rely on the mailserver.
In order to give clients more insights about archive messages being
processed, we're adding this additional signal that informs clients when
the import of downloaded history archive messages has started.
updates
When channels where removed from communities by the community owner,
community members would not remove the chats in their own databases.
See https://github.com/status-im/status-desktop/issues/8000 for more
info.
This commit ensures that nodes delete removed chats from their local
chats table and also deregister the corresponding transport filters.
We don't allow multiple channels with the same name in communities.
Discord allows for multiple channels with the same name (living in
different categories), so this is an error case in our import tool.
This commit improves the user facing error message of this scenario.
We've been limiting the amount of errors being emitted to clients
to reduce payload pressure and also due to the fact that we won't be
rendering more than 3 error items in the UI.
We want to let actual errors through (as opposed to warnings), even if
the limit was reached.
* feat(ActivityCenter): Add community request AC notification
* feat(ActivityCenter): Add CommunityID to AC notification
* feat(ActivityCenter): Add membership status for community membership AC notifications
* feat(ActivityCenter): Add tests for community notifications and fix naming
* Add notification for kicked from community action
* feat(ActivityCenter): Fix for missing notification objects for tests
Prior to this commit we had a `CreateHistoryArchiveTorrent()` API which
takes a `startDate`, an `endDate` and a `partition` to create a bunch of
message archives, given a certain time range.
The function expects the messages to live in the database, which means,
all messages that need to be archived have to be saved there at some
point.
This turns out to be an issue when importing communities from third
party services, where, sometimes, there are several thousands of messages
including attachment payloads, that have to be save to the database
first.
There are only two options to get the messages into the database:
1. Make one write operation with all messages - this slow, takes a long
time and blocks the database until done
2. Create message chunks and perform multiple write operations - this is
also slow, takes long but makes the database a bit more responsive as
it's many smaller operations instead of one big one
Option 2) turned out to not be super feasible either as sometimes,
inserting even a single such message can take up to 10 seconds
(depending on payload)
Which brings me to the third option.
**A third option** is to not store those imported messages as waku
message into the database, just to later query them again to create the
archives, but instead create the archives right away from all the
messages that have been loaded into memory.
This is significantly faster and doesn't block the database.
To make this possible, this commit introduces
a `CreateHistoryArchiveTorrentFromMessages()` API, and
a `CreateHistoryArchiveTorrentFromDB()` API which can be used for
different use cases.
settings
Turns out `UpdateCommunitySettings()` has never worked. Two parameters
where in the wrong order, cause the SQL statement to never find the row
it has to update.
When fetching torrent info after receiving a magnet link,
it can happen that the request times out.
We want to retry downloading the data again at least once more
before giving up
The default logger writes to `geth.log`, which makes debugging
the archive protocol pretty hard.
This adds an additional logger that logs to stdout, while keeping
the default logger intact for production.
Richard Ramos
Pascal Precht
Icaro Motta
Igor Sirotin
frank
Andrea Maria Piana
Jonathan Rainville
Roman Volosovskyi
Omar Basem
Patryk Osmaczko
Parvesh Monu
Jakub Sokołowski
Mikhail Rogachev
Sale Djenic
Pascal Precht
yqrashawn
Michal Iskierko
Samuel Hawksby-Robinson
Ibrahem Khalil
Boris Melnik