Commit Graph

3259 Commits

Author SHA1 Message Date
Boris Melnik af1327f95d
fix(mute): Fix time calculation and skip empty response sending (#3557) 2023-06-01 17:17:42 +03:00
IvanBelyakoff 7adfbb5467
feat(wallet): SequentialFetchStrategy improvements: (#3551)
- reverted a change that stopped looking for ERC20 transfers if no nonce
  and balance change found within a block range for ETH
- implemented sending EventRecentHistoryReady event at a proper time
- moved EventFetchingRecentHistory event to Strategy type as it does not make
sense to send this event in loop
- moved iterating through blocks logic to inside of `loadTransfers` command, which
now accepts a block range.
- reuse `uniqueHeaders` function in commands.go
- clean up

Updates #10246
2023-06-01 16:09:50 +03:00
frank 5d62a9eef4
Fix/mobile issue 16043 (#3560)
* update color hash/id on login

* bump version

* fix test

* fix conflict with LoginAccount
2023-06-01 20:57:05 +08:00
Andrea Maria Piana 7fd9fefdef Add LoginAccount endpoint
This commit adds LoginAccount endpoint.
This makes it consistent with CreateAccount and RestoreAccount as they
use similar config.

The notable difference with the previous endpoint is the API, which is
the same as CreateAccount/RestoreAccount, and the fact that it will
override your networks configuration.

Storing them in the config is now not needed anymore, as that's always
driven from the backend, and we won't allow custom networks in the new
wallet.
2023-06-01 10:05:11 +01:00
Siddarth Kumar d37b605796
adjust padding for QR Images (#3510) 2023-06-01 13:41:01 +05:30
Igor Sirotin a7df4ed388
Local Pairing update for `StopNode` (#3536) 2023-06-01 08:33:57 +03:00
Alex Jbanca 790efc16aa fix(databaseLocks): Fixing database lock errors on transactions
The default transaction lock is `deferred`. This means that the transaction will automatically become read or write transaction based on the first DB operation. In case the first operation is `SELECT` the transaction becomes read transaction, otherwise write transaction. When a read transaction tries to write the DB sqlite will promote the transaction to a write transaction if there is no other transaction that holds a lock. When the promotion fails `database is locked` error is returned. The error is returned immediately and does not use the busy handler.
In our case almost all read transaction would fail with `database is locked` error.

This fix is changing the default transaction lock to `IMMEDIATE`. It translates to `BEGIN IMMEDIATE` instead of `BEGIN`. In this mode, the transaction will be created as a write transaction no matter what DB operation will run as part of the transaction. The write transaction will try to obtain the DB lock immediately when `BEGIN IMMEDIATE` is called and the busy handler is used when the DB is locked by other transaction.

Fixing: https://github.com/status-im/status-desktop/issues/10838
2023-05-31 08:46:26 +02:00
Andrea Maria Piana 46399d1905
Use provided datadir 2023-05-30 23:09:36 +02:00
Jonathan Rainville b701005f3b
fix(pins): prepare content of pin messages so stickers work (#3523) 2023-05-30 15:05:38 -04:00
Richard Ramos b1a6607dbb chore: upgrade libp2p 2023-05-30 10:49:52 -04:00
Richard Ramos 3e4fe1e12f refactor: move circuit relay code to go-waku 2023-05-30 10:49:52 -04:00
Jonathan Rainville b68853c53e
fix(messenger_handler): fix group chat not being set as active (#3545)
Fixes an issue where if a group chat was first received from a non-contact, and later received from a contact, it still wouldn't save it as active.

That's because we checked if we were **newly** added instead of just if we were added. That meant that in the case I described above, the chat would then never have the chance to be set active.
2023-05-30 09:49:46 -04:00
Jonathan Rainville a6285cc827
Fix chats reseting when a Sync signal comes for communities (#3539)
* fix(community): stop re-joining comm when receiving a sync community msg

Fixes an issue with chats being reset. Since joining a community resaves the chats with the synced default value, it resets the sate of the chats, losing the unread messages, the muted state and more.
The solution is to block the re-joining of the community. In the case of the sync, we catch that error and just continue on.

* fix(import): fix HandleImport not saving the chat

Doesn't change much, but it could have caused issues in the future, so since we might have modified the chat, we make sure to save them
Also adds a test

* fix tests
2023-05-29 13:57:05 -04:00
Jakub Sokołowski 589cc965e3
ci: add timeouts to all Jenkinsfiles
This is especially important for tests.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-05-29 12:40:08 +02:00
Anthony Laibe 645cd5d03b feat: return only enabled networks from the reader 2023-05-29 08:37:08 +02:00
Roman Volosovskyi aded83fc6e
Fix panic on receiving nil message at handleSyncKeypair 2023-05-26 18:43:44 +02:00
IvanBelyakoff 963f9e3dbd
fix(wallet): replace POKT rpc urls with archival ones (#3533) 2023-05-26 14:32:59 +03:00
IvanBelyakoff 2dc6700e38
chore(wallet): fix annoying error on missing argument while logging (#3532)
error on token fetch from store
2023-05-26 11:41:33 +03:00
IvanBelyakoff dc84afb751
Add new transfers fetching for sequential strategy (#3520)
* chore(wallet): refactor sequential transfers commands

* feat(desktop/wallet): add fetching new blocks and transfers for
transfers SequentialFetchStrategy

Updates #10246
2023-05-26 11:27:48 +03:00
Sale Djenic 03d9af0b95 feat: `AddressExists` endpoint added 2023-05-26 07:12:40 +02:00
Roman Volosovskyi ee4003279a
Changes necessary to restore wallet on mobile app 2023-05-25 21:08:34 +02:00
Sale Djenic 11408f92d7 test: remove flaky test 2023-05-25 19:46:47 +02:00
Sale Djenic 34f5ef031c feat: a profile keypair name follows display name
As part of this commit `UpdateKeypairName` endpoint added,
will be used to rename all but the profile keypairs.
2023-05-25 19:46:47 +02:00
Sale Djenic 20f38bf62b tests: `GetProfileKeypairForTest` from test helper updated and provides higher granularity 2023-05-25 19:46:47 +02:00
Sale Djenic 0eca7f5088 fix: issues after rebase fixed 2023-05-25 19:46:47 +02:00
Sale Djenic b16a38408a tests: migration test fixed
An issue was that in the `0.97.3` version we didn't have `key_uid` column, later
it was added but there was no chance to set in `key_uid` value properly during
migration and we left it empty. Now in `accounts_and_keycards_improvements.up.sql`
script a constraint `CHECK (length(trim(key_uid)) > 0)` is set for `key_uid` column and
because of it migration test couldn't pass cause `key_uid` was empty.

How it is fixed: the same test account is used but just `key_uid` was added to table to
make migration tests pass again and stored Status test account data refer to `0.132.0`
version.
2023-05-25 19:46:47 +02:00
Sale Djenic 03f93e9f6c tests: tests affected by accounts improvements are fixed 2023-05-25 19:46:47 +02:00
Sale Djenic eeaaf0ce3f feat: accounts improvements applied
- old `accounts` table is moved/mapped to `keypairs` and `keypairs_accounts`
- `keycards` table has foreign key which refers to `keypairs.key_uid`
- `Keypair` introduced as a new type
- api endpoints updated according to this change
2023-05-25 19:46:47 +02:00
Michal Iskierko 1c17fbeacc feat(RemoteBurn): Add remote burn collectibles functionality
* Updating smart contract (new burn function)
* remoteBurn() and estimateRemoteBurn() functions
2023-05-25 13:34:53 +02:00
Lukáš Tinkl f3c826275e
[go-sqlcipher] replace AES implementation with mbedtls (#3518)
HW accelerated versions provided for x86_64 and arm64, with compile time
guards and runtime detection

needed for status-im/status-desktop#10726

---------

Co-authored-by: Alex Jbanca <alexjb@status.im>
2023-05-25 11:30:11 +02:00
Vitaliy Vlasov ddfd0c5b14 Add lc-proxy-wrapper dep 2023-05-24 18:48:16 +03:00
Stefan e78a73bd9f feat(Wallet) complete the filter API
It uses the current data only and doesn't extend with new types or
include new features in activity sources DBs.

Major changes:
- Partially filter by chain IDs
- Partially filter by Status if it is the case
- Partially filter by token types
- Filter by counterparty addresses
- Use wallet accounts for TO/FROM instead of filters

Closes: #10634
2023-05-23 14:48:29 +02:00
Patryk Osmaczko 5777bb429a feat: enable multiple db connections with sqlcipher
part of: status-im/status-desktop#10558
2023-05-23 14:43:42 +02:00
frank 75864528b1
fix: display name/nickname/ens which includes dash is not resolved in the chat when mentioned (#3516) 2023-05-23 18:07:49 +08:00
Mikhail Rogachev 8c90353bc0
fix: create a CR on contact sync with received CR state (#3495)
* fix: create a CR on contact sync with received CR state

* fix: create a CR on contact sync with sent CR state

* Review fixes

* Fix: ignore own contact installation or syncing
2023-05-22 16:22:33 +04:00
Anthony Laibe 8b05ba6d8b fix: check balance for address details 2023-05-22 09:13:09 +02:00
Andrea Maria Piana 959e370389 config: add new Push notifications fleet
Also drop no longer existing `eth.test` fleet.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-05-19 17:29:18 +01:00
Andrea Maria Piana 6acb34bab7 Allow setting up wallet with new config
This commit does a few things:

1) Extend create/import account endpoint to get wallet config, some of
   which has been moved to the backend
2) Set up a loop for retrieving balances every 10 minutes, caching the
   balances
3) Return information about which checks are not passing when trying to
   join a token gated community
4) Add tests to the token gated communities
5) Fixes an issue with addresses not matching when checking for
   permissions

The move to the wallet as a background task is not yet complete, I need
to publish a signal, and most likely I will disable it before merging
for now, as it's currently not used by desktop/mobile, but the PR was
  getting to big
2023-05-19 13:23:21 +01:00
IvanBelyakoff 17aaaf1dca
fix(desktop/wallet): fix bug in balance_cache - balances and nonces (#3509)
were stored in cache by pointers, which caused falsy cache hits in loop
because pointers with same address were created for different block
numbers. Now cache uses block numbers of uint64 as key, which can
overflow but it is not a problem since we use this cache for values
comparison, not as user data.
Fix crash on nil pointer in log.
Remove some unused code.
Protect nonceRanges with mutex while reading.

Updates #10246
2023-05-19 14:46:54 +03:00
IvanBelyakoff 94c7cd32af
feat(wallet): Implemented SequentialFetchStrategy for transfers. (#3480)
By default OnDemandFetchStategy is still used.

Updates #10246
2023-05-19 11:19:48 +03:00
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
frank c7aebfeed3
Fix/mobile issue 15899 : error when trying login under updated password (#3493)
* fix: unable to reset password for newly created account using CreateAccountAndLogin

* remove unnecessary print

* add TestCreateAccountAndLogin

* update TestCreateAccountAndLogin

* bump version
2023-05-18 22:00:55 +08:00
Richard Ramos 0f0b6f7578
feat: add `pprof` flag to spiff-workflow 2023-05-18 15:22:05 +02:00
yqrashawn 23138d0dbb
feat: start media server with no db right after logout (#3505) 2023-05-18 19:22:20 +08:00
yqrashawn 6fed50ce53
feat: media server and colorhash related change (#3500) 2023-05-18 14:27:16 +08:00
Andrea Maria Piana c26f45ae3d
Bump version to 0.151.8 2023-05-17 18:22:50 +01:00
Richard Ramos 32ac1c8d8e fix: peer exchange service is not required for clients 2023-05-17 18:23:27 +01:00
Andrea Maria Piana efb1036429 update go-waku v0.6.0 2023-05-17 18:23:27 +01:00
Alex Jbanca 62e3e9bd62
performance(sqlcipher): provide optimized versions of sha1_compress (#3490)
* performance(sqlcipher): provide optimized versions of sha1_compress
- on ARM, use Neon/ACLE extensions (a 3x-4x perf improvement)
- on x64 use a plain C implementation from nayuki (a 2x-3x perf gain)

Unfortunately, we can't use the dedicated `SHA1` extension on x64 as this
became widely available only recently (esp. on AMD CPUs)
2023-05-16 20:18:38 +02:00
Jonathan Rainville 9151aa7f04
fix(unviewed): fix edge case where unviewed count would be -1 (#3491) (#3496) 2023-05-16 12:11:52 -04:00