Commit Graph

64 Commits

Author SHA1 Message Date
Dario Gabriel Lipicar 54ea0981a5 fix: properly handle tokenID 0 when filtering activity by collectibles 2024-02-23 09:43:20 -03:00
Stefan e9ff0fbefe feat(wallet) add GetMoreForFilterSession API method
Also fix StopFilterSession to always notify client

Updates #12120
2024-02-13 17:37:44 +01:00
Stefan ca973b4aa6 feat(wallet) use sql activity filter for incremental updates
Switch from the prototype of duplicating the SQL filter as a runtime
and keeping them in sync on each event that might invalidate the current
filtered entries to a simpler approach of requesting the filter again
and doing the diff to detect the new changes.

Also add a new reset API to model the new entries design requirements.

The new approach shows less corner-case to handle and follows one source
of truth concept making debugging and future maintenance easier.

Other changes

- Fix pending mocking to work with multiple calls
- Refactor tests to account for the new changes

Updates status-desktop #12120
2024-02-12 19:29:15 +01:00
Stefan 812910f087 feat(wallet): Add session-based activity API for dynamic updates
This commit introduces the first steps towards implementing a session-based activity API to support dynamic updates of the current visualized filter in the wallet activity service. This change is necessary to move away from static paginated filtering, which was previously done in SQL, to a more dynamic approach that can handle updates in real-time.

The main changes include:
- Add basic `EventActivitySessionUpdated` support for pending transactions.
- Added a `TODO.md` file outlining the plan and requirements for dynamic activity updates.
- New session-related API to the `activity.Service`
- `session.go` contains the logic for session management and event processing related to activity updates.
- Add test case for incremental filter updates.

The commit also includes:

- various other minor changes and refactoring to support the new session-based approach.
- Deprecation notices added to the `api.go` file for methods that are no longer used by the status-desktop application.
- Clarification comments added to the `scheduler.go` file regarding replacement policies.

Updates: #12120

ghstack-source-id: a61ef74184
Pull Request resolved: https://github.com/status-im/status-go/pull/4480
2024-02-01 19:55:06 +01:00
Stefan 21e6914a3c fix(wallet) fix reading amount for pending transactions
The reading of the amount for pending transactions was done in the same
way as for transfers table. However, the transfers table has a string
hex representation of the amount, while the pending transactions table
has a binary representation of the amount (*big.Int). This was
triggering the not int warning and value was missing.

Updates status-desktop #12120
2024-02-01 18:28:55 +01:00
Stefan 31a2d403ef fix(wallet) prioritize type SendAT for pending entries in activity
In case both to/from addresses are present in the list we were using
the same logic as for transfers. However, this doesn't make sense given
that we can have only one entry in pending activity.

The following cases are still covered

- When the receiver is in addresses we get received
- When both receiver and sender are in the list will get sent
- When the sender is on the list we will get sent

Updates status-desktop #12120
2024-02-01 16:59:09 +01:00
Ivan Belyakov 7d1927396a fix(wallet): do not return gas-only ETH entries of transfers table for
activities request

Closes #4618
2024-01-30 11:37:53 +01:00
Dario Gabriel Lipicar 959dcbdea5 feat: lazy load collectibles metadata 2023-12-19 21:27:20 -03:00
Cuteivist 79f0d8a5ec
bug: Added missing out chain data (#4303) 2023-11-30 12:37:32 +01:00
Dario Gabriel Lipicar e17d4606b1 fix: implement cancellable collectibles requests 2023-11-17 10:35:30 -03:00
Cuteivist e5fbe40b9a
fix: Show community token mint as Mint tx type (#4214) 2023-10-26 20:39:31 +02:00
Cuteivist 5674a010cd
feat: Handle Status Mint transaction (#4137) 2023-10-12 12:21:03 +02:00
Cuteivist 6798d1ac5c
feat: Handled token mint activity (#4126) 2023-10-11 07:10:08 +02:00
Cuteivist ecc8b4cb55
feat: Wallet activity collectibles model (#4074) 2023-10-03 12:49:04 +02:00
Cuteivist cff96f99e0
Bug: Refresh recipients model (#4076) 2023-10-02 13:46:05 +02:00
Stefan 5c7748dbf7 feat(wallet) add API to cancel current activity filter
Closes status-desktop #11036
2023-09-21 13:56:44 +02:00
Stefan 579f7e4a52 chore(wallet) optimize the filer query
Main changes:

- Use tr_type instead of IN clause
- Use binary (X'...' syntax) directly into the query instead of
  converting DB values to HEX
  - Found to be slightly faster than query parameters in the dedicated benchmark
  - Didn't see much improvement in filter benchmarks
- Tried various combinations of optimizations but without impressive performance results

Benchmark results:

| Name                   | Original   | tr_type   | join     | hex      | no-db     | db_only   |     last |    net_j |
|:-----------------------|:-----------|:----------|:---------|:---------|:----------|:----------|---------:|---------:|
| RAM_NoFilter-10        | 49580229   | 51253242  | 51112462 | 50915133 | 121217817 | 141691008 | 50908642 | 50239712 |
| SSD_NoFilter-10        | 49963604   | 51393588  | 51213038 | 50881483 | 120785679 | 141063467 | 50462767 | 49676867 |
| SSD_MovingWindow-10    | 53695712   | 54155292  | 54161733 | 54061325 | 126966633 | 146866017 | 53479929 | 53350475 |
| SSD_AllAddr_AllTos-10  | 41382804   | 41195225  | 51684175 | 52107262 | 64348100  | 97608833  | 50523529 | 49968321 |
| SSD_OneAddress-10      | 34945275   | 35103850  | 31066429 | 31328762 | 50927300  | 54322971  | 30098529 | 30252546 |
| FilterSend_AllAddr-10  | 39546808   | 37566604  | 38389725 | 38260738 | 114820458 | 125588408 | 37127625 | 36864575 |
| FilterSend_6Addr-10    | 41221458   | 41111225  | 40848288 | 40135492 | 118629700 | 128200467 | 38942521 | 39012100 |
| FilterThreeNetworks-10 | -          | -         | -        | -        | -         | -         | 50058929 | 49854450 |

Update status-desktop: #11036
2023-09-21 13:56:44 +02:00
Cuteivist bc4093299e
feat: Filter by collectibles (#4028) 2023-09-21 08:58:36 +02:00
Cuteivist d29c6c5b6f
feat: Calculate finalize status and filter by it (#3969) 2023-09-20 10:30:31 +02:00
Stefan ad971278d9 chore(wallet) remove activity filter dependency on accounts
Require that the activity filter is passed at least one account address
to extract type sent/received

Updates status-desktop #11980
2023-09-19 09:06:57 +02:00
Stefan 195214765b fix(wallet) fix filter by activity type
The activity type filtering was not stable in relation to addresses
filter which was generating unexpected Send/Receive type in the
corner-case when both sender and receiver was in the address list.

Updates status-desktop #11960
2023-09-12 11:58:24 +02:00
Stefan 70341f85a5 fix(wallet) send/receive for duplicate transactions
Brings consistency in case when sender and receiver are both in the
filter address list. This fixes the case of sender and receiver in
addresses and filters out duplicate entries.

Also

- refactor tests to provide support for owners
- adapt TestGetActivityEntriesWithSameTransactionForSenderAndReceiverInDB
  to the use of owner instead of from
2023-09-12 11:58:24 +02:00
Cuteivist e77fc59f5e
feat: Add activity filtering by contract deploy and minting (#4009) 2023-09-11 11:54:37 +02:00
Stefan 555aae4d0f fix(wallet) filter by sub-transaction chain ID
Closes status-desktop #12077
2023-09-06 21:08:39 +02:00
Stefan c3d2ff0723 chore(wallet) move filter activity sql query in a separate file
This should make the complex query more readable and maintainable
Embed the file content in the complied binary

Updates status-desktop #12077
2023-09-05 22:15:04 +02:00
Stefan c0f32748b4 feat(wallet) lazy load activity info optimization
Trigger async fetching of extra information on each activity filtering
request. Only emit the update event for incomplete entries.

Other changes:

- Make DataEntry light as event payload by making all the fields
  optional
- Add new required fields to the activity DataEntry
- Add collectibles.ManagerInterface to aid testing

Note: this PR keeps compatibility with current master by always
providing non-optional multi-transaction ID. The TODO will be executed
before merging the status-desktop PR.

Experienced a hang on FetchAssetsByCollectibleUniqueID call with:
[{{5 0x21263a042aFE4bAE34F08Bb318056C181bD96D3b} 1209},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 237},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 236},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 832},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 830},
{{5 0x9A95631794a42d30C47f214fBe02A72585df35e1} 853}]

Updates status-desktop #11597
2023-09-01 19:11:14 +02:00
Stefan 71800a19f1 fix(wallet) fix pending transactions notification
Also, add regression test.
2023-09-01 00:18:26 +02:00
Stefan 9cb5e1d3e6 chore(wallet) move computing totalFees to status-go
Updates status-desktop #11597
2023-08-30 13:59:52 +02:00
Stefan 423991fb30 chore(wallet) lookup symbols for wallet activity filter
Updates status-desktop #11597
2023-08-29 17:25:48 +02:00
Cuteivist dd3e408a4e
feat(wallet/activity): Added API for tx and multiTx details (#3939) 2023-08-24 14:23:40 +02:00
Stefan 524c21834b fix(wallet) propagate multi-transactions IDs to transfers
Mainly refactor API to have control on pending_transactions operations.
Use the new API to migrate the multi-transaction ID from to transfers
in one SQL transaction.
The refactoring was done to better mirror the purpose of pending_transactions

Also:
- Externalize TransactionManager from WalletService to be used by
  other services
- Extract walletEvent as a dependency for all services that need to
  propagate events
- Batch chain requests
- Remove unused APIs
- Add auto delete option for clients that fire and forget transactions

Updates status-desktop #11754
2023-08-22 18:39:42 +02:00
Ivan Belyakov ea38b788b3 tests: Added unit test for copying 'keypairs_accounts' table to the wallet db.
Added a unit test for changing app and wallet DBs passwords.
Refactored geth_backend to simplify and allow wallet db password changing.
Fixed opening database with wrong password.
2023-08-18 09:00:56 +02:00
Ivan Belyakov 2df9df10ab fix(tests): moved test db setup to a common place 't/helpers', created
interface for initializing db, which is implemented for appdatabase and
walletdatabase. TBD for multiaccounts DB.
Unified DB initializion for all tests using helpers and new interface.
Reduced sqlcipher kdf iterations for all tests to 1.
2023-08-18 09:00:56 +02:00
Ivan Belyakov d106b449b6 feat(wallet): move wallet-related tables to a dedicated db.
The only place where appDB is used in wallet is activity,
which refers to `keycards_accounts` table. So a temporary
table `keycards_accounts` is created in wallet db and updated
before each activity query.
2023-08-18 09:00:56 +02:00
Dario Gabriel Lipicar d6aae82566 chore: move multi-client scheduler implementation to separate file 2023-08-15 10:17:10 -03:00
Cuteivist 1f510eae70
fix: Query token id for transactions (#3851) 2023-08-04 13:40:36 +02:00
Cuteivist 06c4eb5df9
feat: Handle collectible mint (#3852) 2023-08-04 12:47:22 +02:00
Stefan eb8d74e1ae feat(wallet) add request id to activity events
Add and use the optional chainID in the wallet event structure.

Updates status-desktop #11380
2023-07-31 18:22:13 +02:00
Dario Gabriel Lipicar f138964a9c chore: move scheduler to async dir 2023-07-27 12:12:03 -03:00
Stefan d956a3e854 feat(wallet) chain id multi-transaction filtering
Updates status-desktop #11631
2023-07-26 13:54:55 +02:00
Cuteivist 84bfdf4aab
feat: Add contract deployment tx type (#3770) 2023-07-20 16:04:30 +02:00
Stefan 8926990c2d chore(wallet) retrieve all required activity header data as metadata
Add missing fields to activity metadata.

Updates status-desktop #11173
2023-07-20 11:34:41 +02:00
Stefan 64f7706567 chore(wallet) index activity filter columns to speed up queries
Implemented required benchmarks

Also

- refactor tests as needed
- don't filter temporary or incomplete transfers (transfers.loaded != 1)

Updates status-desktop #11351
2023-07-14 16:11:14 +02:00
Stefan e6705f264c chore(wallet) refactor activity tests 2023-07-14 16:11:14 +02:00
Stefan 9a30674590 feat(wallet) deliver pending changed events to clients
Add wallet events feed to TransactionManager and send pending changed
events on add and delete

Also

- Remove hardcoded values in the filter query
- Small improvement to query

Updates status-desktop #11233
2023-07-07 15:06:14 +02:00
Stefan 0b9555bf31 fix(wallet) returning wrong address identity in filter
This fixes returning address identity after that was changed by
the real to/from fix. Previously the address was wrongly used as from

Updates status-desktop #11233
2023-07-05 10:36:45 +02:00
Stefan 3b0f4ee2c1 fix(wallet) fix GetMultiTransactions for to_amount nil
Updates status-desktop #11233
2023-07-04 16:49:08 +02:00
Stefan 293fa9d248 fix(wallet) fix filtering failing with NULL values in to_address
Added test to validate the fix

Also change JOIN to LEFT JOIN between `multi_transactions` and `transactions` table so that we return all entries from `multi_transactions` table even if there is no matching entry in `transactions` table, which might be the case for multi transactions that that could not be detected. This will postpone the error case until we get into details of the multi transaction.

Updates status-desktop #11233
2023-07-04 15:51:33 +02:00
Stefan c61a4000d8 feat(wallet) make remaining filter apis async
Implement activity.Scheduler to serialize and limit the number of
calls on the activity service. This way we protect form inefficient
parallel queries and easy support async and rate limiting based on the
API requirements.

Refactor the activity APIs async and use the Scheduler for managing
the activity service calls configured with one of the two rules: cancel
ignore.

Updates status-desktop #11170
2023-07-03 16:50:48 +02:00
Stefan f07a79cd18 fix(wallet) fix crash if to_address is NULL in transfers
Add nill tests for TestGetRecipients, GetOldestTimestamp
Also fix returning duplicate addresses in TestGetRecipients

Updates status-desktop #11170
2023-06-28 09:55:43 +02:00