Commit Graph

2669 Commits

Author SHA1 Message Date
Roman Volosovskyi a399c04346
Remove Ropsten and Rinkeby test networks 2022-10-24 14:17:46 +02:00
Pascal Precht 0ccdec2985 feat(CommunitiesManager): introduce `CreateHistoryArchiveTorrentFromMessages` API
Prior to this commit we had a `CreateHistoryArchiveTorrent()` API which
takes a `startDate`, an `endDate` and a `partition` to create a bunch of
message archives, given a certain time range.

The function expects the messages to live in the database, which means,
all messages that need to be archived have to be saved there at some
point.

This turns out to be an issue when importing communities from third
party services, where, sometimes, there are several thousands of messages
including attachment payloads, that have to be save to the database
first.

There are only two options to get the messages into the database:

1. Make one write operation with all messages - this slow, takes a long
   time and blocks the database until done
2. Create message chunks and perform multiple write operations - this is
   also slow, takes long but makes the database a bit more responsive as
   it's many smaller operations instead of one big one

Option 2) turned out to not be super feasible either as sometimes,
inserting even a single such message can take up to 10 seconds
(depending on payload)

Which brings me to the third option.

**A third option** is to not store those imported messages as waku
message into the database, just to later query them again to create the
archives, but instead create the archives right away from all the
messages that have been loaded into memory.

This is significantly faster and doesn't block the database.

To make this possible, this commit introduces
a `CreateHistoryArchiveTorrentFromMessages()` API, and
a `CreateHistoryArchiveTorrentFromDB()` API which can be used for
different use cases.
2022-10-24 13:28:55 +02:00
Pascal Precht 7eb66d09e7 feat(communities): add encryption suppord to archive protocol 2022-10-24 12:35:38 +02:00
Anthony Laibe a69a59c601
feat: add reverse lookup for ens (#2912) 2022-10-24 12:07:44 +02:00
Pascal Precht b2dce92f3f fix(Communities): correct query params order when updating community
settings

Turns out `UpdateCommunitySettings()` has never worked. Two parameters
where in the wrong order, cause the SQL statement to never find the row
it has to update.
2022-10-24 12:02:52 +02:00
Jakub Sokołowski 5b941cab0c
ci: drop unnecessary go get command
Not sure why that was added at some point, but it doesn't seem
necessary, and even if it was `go install` should be used.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2022-10-24 11:26:38 +02:00
Jakub Sokołowski d7e79dba80
go.mod: ran make tidy
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2022-10-24 11:25:08 +02:00
Anthony Laibe 37c9eb1a9d
fix: token override when scanning (#2910) 2022-10-24 08:45:14 +02:00
Andrea Maria Piana bef211b5d6 Send all encryption keys 2022-10-20 12:19:44 +01:00
frank 87560c0aae
support creating an arbitrary number of communities (#2901) 2022-10-20 16:56:29 +08:00
Roman Volosovskyi 5c3435c12f
Changes necessary to make wallet work pokt networks 2022-10-19 10:40:13 +02:00
frank 94fea4725d
replace web3-utils methods usage by status-go (#2840) 2022-10-18 21:36:54 +08:00
Pascal Precht 69e84b5673 feat(communities): retry downloading archive data
When fetching torrent info after receiving a magnet link,
it can happen that the request times out.

We want to retry downloading the data again at least once more
before giving up
2022-10-17 11:33:02 +02:00
Pascal Precht dd49c4c954 feat(communities): add archiveLogger for stdout logs
The default logger writes to `geth.log`, which makes debugging
the archive protocol pretty hard.

This adds an additional logger that logs to stdout, while keeping
the default logger intact for production.
2022-10-14 16:31:21 +02:00
Pascal Precht 616a58f5c9 fix(MessengerResponse): ensure `savedAddresses` response as `omitempty`
Otherwise the json response will be `null` instead of an empty array
(which broke desktop): https://github.com/status-im/status-desktop/pull/7888
2022-10-13 14:53:09 +02:00
Anthony Laibe d9b44777c9
feat: update opensea testnet to use goerli (#2894) 2022-10-12 09:59:38 +02:00
Khushboo-dev-cpp 3364c159f4
feat: Added API for getting historical data for a token (#2882) 2022-10-10 21:02:04 +02:00
Richard Ramos b14de2dd9d fix: content topic in wakuv2 2022-10-10 10:38:42 -05:00
Richard Ramos d4d35710be fix: community encryption was not being marshalled 2022-10-10 13:11:10 +02:00
Stefan b9d6e6d8c8 feat: sync saved addresses
Main changes:
- Extend saved addresses DB with sync info: sync timestamp, update timestamp
and soft removed flag
- Create custom protobuf message payload to sync saved addresses
- Cleanup saved addresses on each start of messenger, by deleting
soft removed older entries
- Sync all saved addresses on Messenger.SyncDevices calls
- Sync particular changes to saved addresses
- Add SavedAddressManager instance to messenger
  - Note, can't find a clean way to pass the SavedAddressManager to the
  messenger, so we create another one
- Add tests for sync and new DB API

Closes: #7229
2022-10-10 11:26:35 +02:00
r4bbit.eth 6c148389c1
fix(message_persistence): ensure query cursor value is supplied where (#2898)
needed

Unfortunately, this one slipped through when introducing helper
functions to retrieve messages.

Sometimes, queries need an additional empty `cursor` value.
2022-10-07 13:38:28 -04:00
Stefan daee116878 fix: compiling chat_messsage.proto due to unscoped ContentType 2022-10-07 12:49:04 +02:00
Patryk Osmaczko 73a45dd58e feat: add `SpectateCommunity` api
- added `SpectateCommunity` endpoint, it is supposed to be used in
  scenarios where we want to "Go to public Community" and see its
  content without joining
- added `spectated` field to `Community`, it means we are observing the
  community and its chats but we are not members

Use case:
https://github.com/status-im/status-desktop/issues/7072#issuecomment-1246560885
2022-10-06 21:21:37 +02:00
Richard Ramos bf04ff3502 fix: waku2 store queries 2022-10-05 18:43:10 -04:00
Omar Basem 938e7987ca
feat: edit/delete/reply for emoji/sticker/audio (#2771)
* feat: edit/delete/reply for emoji/sticker/audio
2022-10-05 21:54:47 +04:00
Samuel Hawksby-Robinson dde0c71cfc Added string identifier for Connection String 2022-10-05 12:58:32 +01:00
Samuel Hawksby-Robinson d7e57fbb66 VERSION bump 2022-10-05 12:20:26 +01:00
Samuel Hawksby-Robinson 61904182c5 check and make keystore if required on receiving device 2022-10-05 12:20:26 +01:00
Pascal Precht 9bb1e75939 fix: ensure discord attachment existence is checked with message ID 2022-10-05 13:17:57 +02:00
Pascal Precht f7faebe775 chore: save discord message should use REPLACE INTO 2022-10-05 13:13:38 +02:00
Pascal Precht ffc9aea24f fix: ensure torrent download does not finish too early 2022-10-05 13:13:16 +02:00
Michał 057c7915f4
chore: use provided pictures visibility options when creating settings db entry (#2867)
When initially creating settings, properties 'initialize profile_pictures_show_to'
and 'profile_pictures_visibility' are set according to the provided Setting object.
2022-10-05 12:53:42 +02:00
Richard Ramos 571ab4c88d fix: use minPeers when initializing relay 2022-10-04 14:31:42 -04:00
Richard Ramos 7272bf0d91 fix: waku2 storenode cursor 2022-10-03 18:49:41 -04:00
Pascal Precht 1eb8a5cc2c feat(MediaServer): add support for discord author and attachment URLs 2022-09-29 16:33:15 +02:00
Pascal Precht b01a861e8e feat: add `DiscordMessageAttachment` types and APIs
This adds a new `DiscordMessageAttachment` type which is part of
`DiscordMessage`. Along with that type, there's also a new database
table for `discord_message_attachments` and corresponding persistence
APIs.

This commit also changes how chat messages are retrieved.
Here's why:

`DiscordMessage` can have multiple `DiscordMessageAttachment`.
A chat message can have a `DiscordMessage`.

Because we're `LEFT JOIN`'ing the discord message attachments into the
chat messages, there's a possibility of multiple rows per message.

Hence, this commit ensures we collect queried discord message
attachments on chat messages.
2022-09-29 11:38:29 +02:00
Pascal Precht c01ad05525 feat(common): add `ThirdPartyID` to `Message` and `protobuf.WakuMessage`
Usually, message IDs are generated by their payload and signature and
in receiving nodes calculated in based on the same data as well.

There's no ID attached to messages in-flight.

This turns out to be a bit of a problem for messages that are being
imported from third party systems like discord, as the conversion
and saving of such messages and handling of their possible assets and
attachments are done in separate steps, which changes the message
payloads after their IDs have been generated.

Hence, we're introducing a `ThirdPartyID` property to `common.Message`
and `protobuf.WakuMessage` so receiving nodes of such messages (via the
archive protocol primarily) can easily detect third party/imported
messages and give them special treatment.
2022-09-29 11:15:47 +02:00
yqrashawn f47cb8572d
feat: delete for me (#2866) 2022-09-28 19:42:17 +08:00
Pascal Precht 81e279a055 chore(community): community itself has admin rights
This might look like a weird requirement at a fist glance.
The reason this is needed, is because some message signals require
admin rights to take effect (e.g. PinMessage).

When messages are imported from third-party services,
translated to status messages, signed by the community, and eventually distributed
via the archive protocol, we need to ensure that messages signed
by the community itself are considered as admin privileges as well,
so they can be correctly replayed into the database.
2022-09-28 13:03:56 +02:00
Richard Ramos b8fd999b54
fix: lint (#2845)
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2022-09-27 18:59:02 -04:00
Richard Ramos bf9abfc350
feat: make kdf iterations configurable when creating a multiaccount (#2796) 2022-09-27 16:27:20 -04:00
Roman Volosovskyi 506921509e
Add effective gas price to tx view 2022-09-27 17:15:56 +02:00
Pascal Precht bca26fbdd9 tests(persistence): add missing test for author payload API 2022-09-27 14:45:58 +02:00
Sale Djenic d89c0c8d9e fix(keycard): convert to keycard action removes keystore and doesn't change the pass properly fixed 2022-09-27 12:13:50 +02:00
Andrea Maria Piana 5668bbaa8e
Bump version to 0.110.2 2022-09-22 10:11:58 +01:00
Samuel Hawksby-Robinson d82c914aa5 Fix if configJSON is empty, NOT not empty 2022-09-21 15:28:31 +01:00
Richard Ramos 965e026cd9
fix: TestImageMessageSharing (#2874) 2022-09-21 09:48:04 -04:00
Sale Djenic 698c32f3e3 chore: `UpdateKeycardUID` function exposed 2022-09-21 15:01:53 +02:00
Patryk Osmaczko e4ca8a256c fix: remove ourselves from members when leaving community
Otherwise clients see invalid members count in invitation bubbles.
2022-09-21 08:35:52 +02:00
Pascal Precht eb4ab9316c fix(communities): ensure existing archives are seeded on startup 2022-09-19 15:34:29 +02:00