Commit Graph

237 Commits

Author SHA1 Message Date
Andrea Maria Piana df2a5d8e21 Update go waku & Shorten shutdown 2023-07-17 14:44:32 +01:00
Dario Gabriel Lipicar e6711c8cc8 feat(wallet): detect ETH L1 to L2 bridge transactions 2023-07-10 09:59:05 -03:00
Andrea Maria Piana 67050429df Bump go-waku 2023-07-07 14:49:40 +01:00
Andrea Maria Piana c3ae0e98fe Enable lightclient 2023-07-07 14:49:40 +01:00
Mikhail Rogachev 511d6bfc54
feat: add parsing for new links format (#3665)
* feat(share-links): Add protobuf and encode/decode url data methods

* feat(new-links-format): Adds generators for new links format

* feat: add parsing for new links format

* feat: add messenger-level pubkey serialization and tests

* feat: fix and test CreateCommunityURLWithChatKey

* feat: impl and test parseCommunityURLWithChatKey

* feat: fix and test CreateCommunityURLWithData

* feat:  impl and test parseCommunityURLWithData (not working)

* feat: UrlDataResponse as response share urls api

* feat: impl& tested ShareCommunityChannelURLWithChatKey

* feat: impl & tested ParseCommunityChannelURLWithChatKey

* fix: bring urls to new format

* feat: add regexp for community channel urls

* feat: impl & test contact urls with chatKey, Ens and data

* fix: encodeDataURL/encodeDataURL patch from Samyoul

* fix: fix unmarshalling protobufs

* fix: fix minor issues, temporary comment TestParseUserURLWithENS

* fix: allow url to contain extra `#` in the signature

* fix: check signatures with SigToPub

* chore: lint fixes

* fix: encode the signature

* feat: Check provided channelID is Uuid

* fix(share-community-url): Remove if community encrypted scope

* fix: review fixes

* fix: use proto.Unmarshal instead of json.Marshal

* feat(share-urls): Adds TagsIndices to community data

* feat: support tag indices to community url data

---------

Co-authored-by: Boris Melnik <borismelnik@status.im>
2023-07-04 17:48:52 +04:00
Stefan c61a4000d8 feat(wallet) make remaining filter apis async
Implement activity.Scheduler to serialize and limit the number of
calls on the activity service. This way we protect form inefficient
parallel queries and easy support async and rate limiting based on the
API requirements.

Refactor the activity APIs async and use the Scheduler for managing
the activity service calls configured with one of the two rules: cancel
ignore.

Updates status-desktop #11170
2023-07-03 16:50:48 +02:00
Richard Ramos 28229faec0 chore: bump libp2p related dependencies 2023-06-30 14:23:08 -04:00
Andrea Maria Piana d91d11a01c Call before dispatch for group chat messages 2023-06-29 13:04:49 +01:00
Andrea Maria Piana 6e19098dee Run tests separately, make sure messenger is shutdown, fix torrent test 2023-06-23 11:30:04 +01:00
Andrea Maria Piana 8015cc3e3b Handle race condition when sending envelopes
It happens that an envelope is sent before it's tracked, resulting in
long delays before the envelope is marked as sent.
This commit changes the behavior of the code so that order is now
irrelevant.
2023-06-15 11:28:19 +01:00
Pascal Precht 1a2ca21070 Add chainIds to revealed accounts in memberhsip requests
This commit does a few things:

- Adds a migration that adds chainids to communities_request_to_join_revealed_addresses
- Removes RevealedAddress in favor of RevealedAccount which is now a struct that contains the revealed address, as well as the signature and a list of chain IDs on which to check for user funds
- Changes the logic of sending requests to join a community, such that after creating address signatures, the user node will also check which of the addresses has funds on which networks for the community's token permissions, and add the chainds to the RevealedAccount
- Updates checkPermissionToJoin() such that only relevant chainids are used when checking user's funds. Chain IDs are retrieved from RevealedAccounts and matched against token permission criteria chain IDs
2023-06-12 10:49:29 +02:00
Richard Ramos e53c2c0a6d chore: bump go-waku 2023-06-09 12:54:22 -04:00
Alex Jbanca e8c4b7647f
chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5 (#3559)
* chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5

Changes:
### github.com/mutecomm/go-sqlcipher
1. The improved crypto argorighms from go-sqlcipher v3 are merged in v4
	Tags:
	v4.4.2-status.1 - merge `burn_stack` improvement
	v4.4.2-status.2 - merge `SHA1` improvement
	v4.4.2-status.4- merge 'AES' improvement
2. Fixed `go-sqlcipher` to support v3 database in compatibility mode (`sqlcipher` already supports this) (Tag: v4.4.2-status.3)
3. Upgrade `sqlcipher` to v5.4.5 (Tag: v4.5.4-status.1)

### github.com/status-im/migrate/v4
1. Upgrade `go-sqlcipher` version in `github.com/status-im/migrate/v4`

### status-go
1. Upgrade `go-sqlcipher` and `migrate` modules in status-go
2. Configure the DB connections to open the DB in v3 compatibility mode

* chore(upgradeSQLCipher): Use sqlcipher v3 configuration to encrypt a plain text database

* chore(upgradeSQLCipher): Scanning NULL BLOB value should return nil

Fixing failing tests: TestSyncDeviceSuite/TestPairingSyncDeviceClientAsReceiver;  TestSyncDeviceSuite/TestPairingSyncDeviceClientAsSender

Considering the following configuration:
1. Table with BLOB column has 1 NULL value
2. Query the value
3. Rows.Scan(&dest sql.NullString)

Expected: dest.Valid == false; dest.String == nil
Actual: dest.Valid == true; dest.String == ""

* chore: Bump go-sqlcipher version to include NULL BLOB fix
2023-06-07 08:58:01 +03:00
Patryk Osmaczko 15d2b4fe80 feat: add configurable throttling mechanism for importing msgs
part of: status-im/status-desktop#10815
2023-06-06 08:49:02 +02:00
Richard Ramos b1a6607dbb chore: upgrade libp2p 2023-05-30 10:49:52 -04:00
Richard Ramos 3e4fe1e12f refactor: move circuit relay code to go-waku 2023-05-30 10:49:52 -04:00
Lukáš Tinkl f3c826275e
[go-sqlcipher] replace AES implementation with mbedtls (#3518)
HW accelerated versions provided for x86_64 and arm64, with compile time
guards and runtime detection

needed for status-im/status-desktop#10726

---------

Co-authored-by: Alex Jbanca <alexjb@status.im>
2023-05-25 11:30:11 +02:00
Vitaliy Vlasov ddfd0c5b14 Add lc-proxy-wrapper dep 2023-05-24 18:48:16 +03:00
Andrea Maria Piana 959e370389 config: add new Push notifications fleet
Also drop no longer existing `eth.test` fleet.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2023-05-19 17:29:18 +01:00
Icaro Motta 6fa8c11382
URL unfurling (initial implementation) (#3471)
This is the initial implementation for the new URL unfurling requirements. The
most important one is that only the message sender will pay the privacy cost for
unfurling and extracting metadata from websites. Once the message is sent, the
unfurled data will be stored at the protocol level and receivers will just
profit and happily decode the metadata to render it.

Further development of this URL unfurling capability will be mostly guided by
issues created on clients. For the moment in status-mobile:
https://github.com/status-im/status-mobile/labels/url-preview

- https://github.com/status-im/status-mobile/issues/15918
- https://github.com/status-im/status-mobile/issues/15917
- https://github.com/status-im/status-mobile/issues/15910
- https://github.com/status-im/status-mobile/issues/15909
- https://github.com/status-im/status-mobile/issues/15908
- https://github.com/status-im/status-mobile/issues/15906
- https://github.com/status-im/status-mobile/issues/15905

### Terminology

In the code, I've tried to stick to the word "unfurl URL" to really mean the
process of extracting metadata from a website, sort of lower level. I use "link
preview" to mean a higher level structure which is enriched by unfurled data.
"link preview" is also how designers refer to it.

### User flows

1. Carol needs to see link previews while typing in the chat input field. Notice
   from the diagram nothing is persisted and that status-go endpoints are
   essentially stateless.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_getTextURLs
  Server-->>Client: Normalized URLs
  Client->>Client: Render cached unfurled URLs
  Client->>Server: Unfurl non-cached URLs.\nCall wakuext_unfurlURLs
  Server->>Website: Fetch metadata
  Website-->>Server: Metadata (thumbnail URL, title, etc)
  Server->>Website: Fetch thumbnail
  Server->>Website: Fetch favicon
  Website-->>Server: Favicon bytes
  Website-->>Server: Thumbnail bytes
  Server->>Server: Decode & process images
  Server-->>Client: Unfurled data (thumbnail data URI, etc)
#+end_src
```

```
     ,------.                                 ,------.                             ,-------.
     |Client|                                 |Server|                             |Website|
     `--+---'                                 `--+---'                             `---+---'
        |        Call wakuext_getTextURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |             Normalized URLs            |                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
        |                                        |                                     |
        |----.                                   |                                     |
        |    | Render cached unfurled URLs       |                                     |
        |<---'                                   |                                     |
        |                                        |                                     |
        |         Unfurl non-cached URLs.        |                                     |
        |         Call wakuext_unfurlURLs        |                                     |
        | --------------------------------------->                                     |
        |                                        |                                     |
        |                                        |            Fetch metadata           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        | Metadata (thumbnail URL, title, etc)|
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Fetch thumbnail           |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Fetch favicon            |
        |                                        | ------------------------------------>
        |                                        |                                     |
        |                                        |            Favicon bytes            |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |           Thumbnail bytes           |
        |                                        | <- - - - - - - - - - - - - - - - - -
        |                                        |                                     |
        |                                        |----.                                |
        |                                        |    | Decode & process images        |
        |                                        |<---'                                |
        |                                        |                                     |
        | Unfurled data (thumbnail data URI, etc)|                                     |
        | <- - - - - - - - - - - - - - - - - - - -                                     |
     ,--+---.                                 ,--+---.                             ,---+---.
     |Client|                                 |Server|                             |Website|
     `------'                                 `------'                             `-------'
```

2. Carol sends the text message with link previews in the RPC request
   wakuext_sendChatMessages. status-go assumes the link previews are good
   because it can't and shouldn't attempt to re-unfurl them.

```
#+begin_src plantuml :results verbatim
  Client->>Server: Call wakuext_sendChatMessages
  Server->>Server: Transform link previews to\nbe proto-marshalled
  Server->DB: Write link previews serialized as JSON
  Server-->>Client: Updated message response
#+end_src
```

```
     ,------.                       ,------.                                  ,--.
     |Client|                       |Server|                                  |DB|
     `--+---'                       `--+---'                                  `+-'
        | Call wakuext_sendChatMessages|                                       |
        | ----------------------------->                                       |
        |                              |                                       |
        |                              |----.                                  |
        |                              |    | Transform link previews to       |
        |                              |<---' be proto-marshalled              |
        |                              |                                       |
        |                              |                                       |
        |                              | Write link previews serialized as JSON|
        |                              | -------------------------------------->
        |                              |                                       |
        |   Updated message response   |                                       |
        | <- - - - - - - - - - - - - - -                                       |
     ,--+---.                       ,--+---.                                  ,+-.
     |Client|                       |Server|                                  |DB|
     `------'                       `------'                                  `--'
```

3. The message was sent over waku and persisted locally in Carol's device. She
   should now see the link previews in the chat history. There can be many link
   previews shared by other chat members, therefore it is important to serve the
   assets via the media server to avoid overloading the ReactNative bridge with
   lots of big JSON payloads containing base64 encoded data URIs (maybe this
   concern is meaningless for desktop). When a client is rendering messages with
   link previews, they will have the field linkPreviews, and the thumbnail URL
   will point to the local media server.

```
 #+begin_src plantuml :results verbatim
   Client->>Server: GET /link-preview/thumbnail (media server)
   Server->>DB: Read from user_messages.unfurled_links
   Server->Server: Unmarshal JSON
   Server-->>Client: HTTP Content-Type: image/jpeg/etc
 #+end_src
```

```
     ,------.                                    ,------.                                  ,--.
     |Client|                                    |Server|                                  |DB|
     `--+---'                                    `--+---'                                  `+-'
        | GET /link-preview/thumbnail (media server)|                                       |
        | ------------------------------------------>                                       |
        |                                           |                                       |
        |                                           | Read from user_messages.unfurled_links|
        |                                           | -------------------------------------->
        |                                           |                                       |
        |                                           |----.                                  |
        |                                           |    | Unmarshal JSON                   |
        |                                           |<---'                                  |
        |                                           |                                       |
        |     HTTP Content-Type: image/jpeg/etc     |                                       |
        | <- - - - - - - - - - - - - - - - - - - - -                                        |
     ,--+---.                                    ,--+---.                                  ,+-.
     |Client|                                    |Server|                                  |DB|
     `------'                                    `------'                                  `--'
```

### Some limitations of the current implementation

The following points will become separate issues in status-go that I'll work on
over the next couple weeks. In no order of importance:

- Improve how multiple links are fetched; retries on failure and testing how
  unfurling behaves around the timeout limits (deterministically, not by making
  real HTTP calls as I did). https://github.com/status-im/status-go/issues/3498
- Unfurl favicons and store them in the protobuf too.
- For this PR, I added unfurling support only for websites with OpenGraph
  https://ogp.me/ meta tags. Other unfurlers will be implemented on demand. The
  next one will probably be for oEmbed https://oembed.com/, the protocol
  supported by YouTube, for example.
- Resize and/or compress thumbnails (and favicons). Often times, thumbnails are
  huge for the purposes of link previews. There is already support for
  compressing JPEGs in status-go, but I prefer to work with compression in a
  separate PR because I'd like to also solve the problem for PNGs (probably
  convert them to JPEGs, plus compress them). This would be a safe choice for
  thumbnails, favicons not so much because transparency is desirable.
- Editing messages is not yet supported.
- I haven't coded any artificial limit on the number of previews or on the size
  of the thumbnail payload. This will be done in a separate issue. I have heard
  the ideal solution may be to split messages into smaller chunks of ~125 KiB
  because of libp2p, but that might be too complicated at this stage of the
  product (?).
- Link preview deletion.
- For the moment, OpenGraph metadata is extracted by requesting data for the
  English language (and fallback to whatever is available). In the future, we'll
  want to unfurl by respecting the user's local device language. Some websites,
  like GoDaddy, are already localized based on the device's IP, but many aren't.
- The website's description text should be limited by a certain number of
  characters, especially because it's outside our control. Exactly how much has
  not been decided yet, so it'll be done separately.
- URL normalization can be tricky, so I implemented only the basics to help with
  caching. For example, the url https://status.im and HTTPS://status.im are
  considered identical. Also, a URL is considered valid for unfurling if its TLD
  exists according to publicsuffix.EffectiveTLDPlusOne. This was essential,
  otherwise the default Go url.Parse approach would consider many invalid URLs
  valid, and thus the server would waste resources trying to unfurl the
  unfurleable.

### Other requirements

- If the message is edited, the link previews should reflect the edited text,
  not the original one. This has been aligned with the design team as well.
- If the website's thumbnail or the favicon can't be fetched, just ignore them.
  The only mandatory piece of metadata is the website's title and URL.
- Link previews in clients should be generated in near real-time, that is, as
  the user types, previews are updated. In mobile this performs very well, and
  it's what other clients like WhatsApp, Telegram, and Facebook do.

### Decisions

- While the user typing in the input field, the client is constantly (debounced)
  asking status-go to parse the text and extract normalized URLs and then the
  client checks if they're already in its in-memory cache. If they are, no RPC
  call is made. I chose this approach to achieve the best possible performance
  in mobile and avoid the whole RPC overhead, since the chat experience is
  already not smooth enough. The mobile client uses URLs as cache keys in a
  hashmap, i.e. if the key is present, it means the preview is readily available
  (naive, but good enough for now). This decision also gave me more flexibility
  to find the best UX at this stage of the feature.
- Due to the requirement that users should be able to see independent loading
  indicators for each link preview, when status-go can't unfurl a URL, it
  doesn't return it in the response.
- As an initial implementation, I added the BLOB column unfurled_links to the
  user_messages table. The preview data is then serialized as JSON before being
  stored in this column. I felt that creating a separate table and the related
  code for this initial PR would be inconvenient. Is that reasonable to you?
  Once things stabilize I can create a proper table if we want to avoid this
  kind of solution with serialized columns.
2023-05-18 15:43:06 -03:00
Andrea Maria Piana efb1036429 update go-waku v0.6.0 2023-05-17 18:23:27 +01:00
Alex Jbanca 62e3e9bd62
performance(sqlcipher): provide optimized versions of sha1_compress (#3490)
* performance(sqlcipher): provide optimized versions of sha1_compress
- on ARM, use Neon/ACLE extensions (a 3x-4x perf improvement)
- on x64 use a plain C implementation from nayuki (a 2x-3x perf gain)

Unfortunately, we can't use the dedicated `SHA1` extension on x64 as this
became widely available only recently (esp. on AMD CPUs)
2023-05-16 20:18:38 +02:00
Alex Jbanca ac837365be performance(sqlcipher): Fix burn_stack performance issue
Implements a cross-platform version of https://github.com/OP-TEE/optee_os/pull/3102

1. Remove recursion
2. Use memset instead of while loop
2023-05-12 09:08:39 +03:00
Richard Ramos acad6e4958 chore: bump go-waku 2023-05-09 17:26:29 -04:00
Sale Djenic 31144ed5a3 chore: `password` removed from the `DeleteAccount` function
Asking for a password when removing a key is not necessary.
2023-05-09 10:48:44 +02:00
frank 272173c939
feat:support backup/sync social settings (#3407)
* feat:support backup/sync social settings

* update binddata.go

* fix lint issue

* addressed feedback from @saledjenic @osmaczko

* addressed feedback from @saledjenic @osmaczko

* add corresponding test for local pairing

* addressed feedback from @saledjenic
2023-04-20 06:59:09 +08:00
Samuel Hawksby-Robinson 17bc1e9f5d Rough initial proof of concept for udp network checking 2023-04-17 12:01:38 +01:00
frank 5fd9e93e9c
backup/sync if seed phrase removed (#3385) (#3384)
backup/sync bio
2023-04-13 08:45:25 +08:00
RichΛrd ba1ba1ac02
chore: use go 1.19 and bump go-libp2p (#3371)
* chore: use go 1.19 and bump go-libp2p

* chore: bump golangci-lint

* chore: update go version in shell.nix
2023-04-07 20:23:07 +02:00
frank fd6073ab2e
rewrite mention in go (#3309) 2023-04-07 16:47:38 +08:00
Richard Ramos a6d33b9912 chore: bump go-waku to fix mem leak 2023-03-27 10:56:23 -04:00
Samuel Hawksby-Robinson 7cd7430d31
Improved Local Pairing Separation of Concerns (#3248)
* Moved all configs into config.go

* Completed build out of new config structures

* Completed SenderClient process flow

* Completed sync data Mounter and client integration

* Completed installation data Mounter and client integration

* House keeping, small refactor to match conventions.

PayloadEncryptor is passed by value and used as a pointer to the instance value and not a shared pointer.

* Reintroduced explicit Mounter field type

* Completed ReceiverClient structs and flows

* Finished BaseClient function parity with old acc

* Integrated new Clients into tests

Solved some test breaks caused by encryptors sharing pointers to their managed payloads

* Built out SenderServer and ReceiverServer structs

With all associated functions and integrated with endpoints.

* Updated tests to handle new Server types

* Added docs and additional refinement

* Renamed some files to better match the content of those files

* Added json tags to config fields that were missing explicit tags.

* fix tests relating to payload locking

* Addressing feedback from @ilmotta

* Addressed feedback from @qfrank
2023-03-23 11:44:15 +00:00
Andrea Maria Piana 44a0f5b74d
Add collapsed community categories 2023-03-14 17:13:21 +00:00
Ibrahem Khalil cd96f557f9
Allow underscores for status-tags (#3286) 2023-03-14 17:31:15 +02:00
Richard Ramos cbb016ac00 feat: update go-waku to introduce meta field and deterministic hashing 2023-03-13 21:03:34 -04:00
Richard Ramos b8d5455904 chore: bump go-waku version 2023-03-06 17:39:59 -04:00
Richard Ramos beeef4c5e7 fix: reduce log level for closing conn messages 2023-03-02 14:39:23 -04:00
Anthony Laibe 60e1978bb5 feat: fallback rpc endpoint 2023-02-28 08:43:57 +01:00
Andrea Maria Piana f77bff6d25 Add primary & secondary names 2023-02-27 15:13:43 +00:00
Richard Ramos 665bc8452c fix: panic - putting the same entry twice not supported 2023-02-24 13:29:58 -04:00
RichΛrd 5773616757
fix: set a limit to the multiaddr field of enr (#3230) 2023-02-24 08:24:54 -04:00
RichΛrd 0babdad17b
chore: upgrade go-waku to v0.5 (#3213)
* chore: upgrade go-waku to v0.5
* chore: add println and logs to check what's being stored in the enr, and preemptively delete the multiaddr field (#3219)
* feat: add wakuv2 test (#3218)
2023-02-22 17:58:17 -04:00
dlipicar 1d1a95091d
feat: implement currency formatting calculation (#3191) 2023-02-17 11:11:07 -03:00
RichΛrd f6b4721c4a
fix: invalid length of multiaddr field (but loaded succesfully anyway) (#3185)
* fix: invalid length of multiaddr field (but loaded succesfully anyway)
* chore: update VERSION
2023-02-13 10:41:19 -04:00
Anthony Laibe 51f99a2631 chore: update go-ethereum 2023-02-13 12:36:46 +01:00
Richard Ramos b8dbbe322c fix: bump go-waku to handle cases where enr has less than 1 character 2023-02-10 12:34:24 +00:00
RichΛrd d2cce5ee80
fix: ignore enr multiaddr field if empty (#3175) 2023-02-09 10:06:07 -04:00
Andrea Maria Piana 9a598cc65b Bump go waku 2023-02-08 17:48:09 +00:00
RichΛrd 5b642d3bed
fix: enr exceeds 300 bytes (#3171) 2023-02-07 10:33:26 -04:00
Richard Ramos 03c61e8847 chore: bump go-waku to also log output from go-libp2p into geth.log 2023-02-02 16:26:25 -04:00