feat(wallet) chain id multi-transaction filtering

Updates status-desktop #11631

services/wallet/activity/TODO.md

@@ -1,161 +0,0 @@
-# DB extension and refactoring for activity
-
-## Work in progress
-
-|               | Buy            | Swap           | Bridge        | Send/Receive  |
-| ------------- | -------------- | -------------- | ------------- | ------------- |
-| Activity data | ~~API~~ ~~DB~~ | ~~API~~   DB   |  _API_   DB   |  _API_   DB   |
-| Raw data      | ~~API~~ ~~DB~~ | ~~API~~   DB   |  _API_   DB   |  _API_   DB   |
-| Pending data  | ~~API~~ ~~DB~~ | ~~API~~   DB   |  _API_   DB   |  _API_   DB   |
-
-Legend:
-
-- ~~API~~ - not much or at all provided
-- _API_ - partially provided
-- API - complete
-
-### Summary
-
-Improve on the identified limitations
-
-- [x] Missing filtering data
-- [x] Missing cached (not extracted as a column)
-  - Extracting the data from the raw data is expensive but might be negligible given that usually we should not expect more than 20 entries per second in the worst case scenario.
-- [x] Table extensions
-  - ~~Activity specific info in activity data store (multi_transaction table)~~
-  - Activity specific info in in the transactions data store (transfers table)
-
-### Missing data
-
-Filter requirements
-
-- [x] Activity operation status
-  - [x] `pending`: have to aggregate for `Buy`, `Swap`, `Bridge`
-    - already there for `Send`, `Receive`
-  - [x] `complete`: only extract and check for `status` in the `receipt` for `Send`, `Receive`
-    - For complex operations aggregate the `complete` status `Buy`, `Swap`, `Bridge`
-  - [x] `finalized`: similar to `complete`  for `Send`, `Receive`
-    - all sub-transactions are `complete` for `Buy`, `Swap`, `Bridge`
-  - [x] `failed`: extract from `status` for all sub-transactions
-- [x] `chainID`: aggregate data for activity entries `Bridge`, `Buy`, `Swap`
-- [x] `tokenCode` for activity entries `Send`, `Receive`
-  - For `Bridge` its already there and `Buy`, `Swap` is coming soon
-- [ ] `collectibles`: require adding collectible attributes to activity data (`token_address` and `tokenId`)
-
-UX requirements
-
-- [x] `status`: for status icon and label
-- [ ] `chainIDs`: for chain icons
-  - Missing for `Bridge`, `Buy`, `Swap`
-- [x] `amount`s: add to the activity.Entry
-  - already in DB
-- [x] `tokenCode`s: add to the activity.Entry
-  - already in DB
-- [ ] `to`/`from`/`owner`: add to the activity.Entry
-  - already in DB, coming soon
-- [ ] `tokenIdentity`: collectible is missing (`chainId`, `address`, `tokenId`)
-  - `tokenCode` should be covering fungible operations
-- [x] `identity`: for all the sources
-- [x] `type`: for the main icon and label
-- [x] `time`: timestamp
-
-### Refactoring
-
-Extend `entry.nim:ActivityEntry` and `activity.go:Entry` with presentation layer data
-
-- [x] `activityType`: instead of the current `MultiTransactionType`
-- [x] `status`: for status icon and label
-
-## Current state
-
-### Transfers Table
-
-The `transfers` transactions raw data
-
-- Transaction identity: `network_id`, `hash`, `address`
-  - Implementation by `sqlite_autoindex_transfers_1` unique index
-- `multi_transaction_id`: `multi_transaction` entries to `transfers` entries mapping (one to many)
-- Raw data:
-  - `tx` transaction
-  - `receipt`: transfer receipt
-
-### Multi-Transaction Table
-
-Represented by `multi_transaction`
-
-Responsibilities
-
-- UX metadata for transactions originating from our APP.
-  - `from_address`, `to_address`
-  - `from_asset`, `to_asset`
-  - `from_amount`, `to_amount` (token codes)
-  - `type` identifies the type (initially only Send and Bridge)
-  - `timestamp` the timestamp of the execution
-- Multi-transaction to sub-transaction mapping
-  - The `multi_transaction_id` in the `transfers` and `pending_transaction` table corresponds to the `ROWID` in the `multi_transactions`.
-
-### Pending Transactions Table
-
-The `pending_transactions` table represents transactions initiated from the app
-
-- Transaction identity
-  - `network_id`, `hash`
-    - implemented by the `sqlite_autoindex_pending_transactions_1` index
-    - Note how this is different from the `transfers` table, where the `address` is also part of the identity.
-- `timestamp`: The timestamp of the pending transaction.
-- `multi_transaction_id`: `multi_transaction` entries to `pending_transactions` entries mapping (one to many)
-
-### Schema
-
-Relationships between the tables
-
-```mermaid
-erDiagram
-    multi_transaction ||--o{ transfers : has
-    multi_transaction ||--o{ pending_transactions : has
-    transfers {
-        network_id BIGINT
-        hash VARCHAR
-        timestamp BIGINT
-        multi_transaction_id INT
-        tx BLOB
-        receipt BLOB
-        log BLOB
-    }
-    multi_transaction {
-      ROWID INT
-      type VARCHAR
-    }
-    pending_transactions {
-        network_id BIGINT
-        hash VARCHAR
-        timestamp INT
-        multi_transaction_id INT
-    }
-```
-
-### Dropped tasks
-
-Dropped the DB refactoring and improvements after further discussion and concerns
-
-- [x] Terminology proposal
-  - [x] using `transactions` instead of `transfers` for the raw data
-  - [x] using `activity` instead of `multi-transaction` to better match the new requirements
-  - [x] Convert JSON blobs into structured data
-
-Dropped benchmark performance and move on using theoretical knowledge by adding indexes for what we know only
-Will leave the performance concerns for the next milestone
-
-- [ ] Joining DBs
-- [ ] One activity DB for all require metadata
-  - Pros:
-    - Faster to query (don't know the numbers)
-    - Simpler query will decrease maintenance
-  - Cons:
-    - have to migrate all data, extract and fill the activity on every download of updates for all activities
-- [ ] Keep only filter specific metadata in the Activity DB
-  - Pros:
-    - Less changes to migrate existing data. Still have to maintain activity filtering specific data
-  - Cons:
-    - Slower to query (don't know how much yet)
-    - Complex query increases maintenance

services/wallet/activity/activity.go

@@ -246,7 +246,6 @@ const (
 	fromTrType = byte(1)
 	toTrType   = byte(2)
 
-	// TODO: Multi-transaction network information is missing in filtering
 	// TODO optimization: consider implementing nullable []byte instead of using strings for addresses
 	//                      or insert binary (X'...' syntax) directly into the query
 	//
@@ -322,21 +321,42 @@ const (
 			SELECT
 				multi_transaction_id,
 				MIN(status) AS min_status,
-              COUNT(*) AS count
+				COUNT(*) AS count,
+				network_id
 			FROM
 				transfers
 			WHERE transfers.loaded == 1
 				AND transfers.multi_transaction_id != 0
 			GROUP BY transfers.multi_transaction_id
 		),
+		tr_network_ids AS (
+			SELECT
+				multi_transaction_id
+			FROM
+				transfers
+			WHERE transfers.loaded == 1
+				AND transfers.multi_transaction_id != 0
+				AND network_id IN filter_networks
+			GROUP BY transfers.multi_transaction_id
+		),
 		pending_status AS (
 			SELECT
 				multi_transaction_id,
-              COUNT(*) AS count
+				COUNT(*) AS count,
+				network_id
 			FROM
 				pending_transactions
 			WHERE pending_transactions.multi_transaction_id != 0
 			GROUP BY pending_transactions.multi_transaction_id
+		),
+		pending_network_ids AS (
+			SELECT
+				multi_transaction_id
+			FROM
+			pending_transactions
+			WHERE pending_transactions.multi_transaction_id != 0
+				AND pending_transactions.network_id IN filter_networks
+			GROUP BY pending_transactions.multi_transaction_id
 		)
 	SELECT
 		transfers.hash AS transfer_hash,
@@ -409,7 +429,8 @@ const (
 			OR (HEX(transfers.tx_to_address) IN filter_to_addresses)
 		)
 		AND (includeAllTokenTypeAssets OR (transfers.type = "eth" AND ("ETH" IN assets_token_codes))
-						OR (transfers.type = "erc20" AND ((transfers.network_id, HEX(transfers.token_address)) IN assets_erc20)))
+			OR (transfers.type = "erc20" AND ((transfers.network_id, HEX(transfers.token_address)) IN assets_erc20))
+		)
 		AND (includeAllNetworks OR (transfers.network_id IN filter_networks))
 		AND (filterAllActivityStatus OR ((filterStatusCompleted OR filterStatusFinalized) AND transfers.status = 1)
 			OR (filterStatusFailed AND transfers.status = 0)
@@ -527,8 +548,20 @@ const (
 			OR (multi_transactions.from_asset != '' AND (UPPER(multi_transactions.from_asset) IN assets_token_codes))
 			OR (multi_transactions.to_asset != '' AND (UPPER(multi_transactions.to_asset) IN assets_token_codes))
 		)
-        AND (filterAllActivityStatus OR ((filterStatusCompleted OR filterStatusFinalized) AND agg_status = statusSuccess)
+		AND (filterAllActivityStatus
-        OR (filterStatusFailed AND agg_status = statusFailed) OR (filterStatusPending AND agg_status = statusPending))
+			OR ((filterStatusCompleted OR filterStatusFinalized) AND agg_status = statusSuccess)
+			OR (filterStatusFailed AND agg_status = statusFailed) OR (filterStatusPending AND agg_status = statusPending)
+		)
+		AND (includeAllNetworks
+			OR (multi_transactions.from_network_id IN filter_networks)
+			OR (multi_transactions.to_network_id IN filter_networks)
+			OR (multi_transactions.from_network_id IS NULL
+				AND multi_transactions.to_network_id IS NULL
+				AND (EXISTS (SELECT 1 FROM tr_network_ids WHERE multi_transactions.ROWID = tr_network_ids.multi_transaction_id)
+					OR EXISTS (SELECT 1 FROM pending_network_ids WHERE multi_transactions.ROWID = pending_network_ids.multi_transaction_id)
+				)
+			)
+		)
 
 	ORDER BY timestamp DESC
 	LIMIT ? OFFSET ?`

services/wallet/activity/activity_test.go

@@ -127,9 +127,6 @@ func fillTestData(t *testing.T, db *sql.DB) (td testData, fromAddresses, toAddre
 	td.multiTx1Tr1 = trs[2]
 	td.multiTx1Tr2 = trs[4]
 
-	// TODO: This got automatically set by GenerateTestTransfers to USDC/Mainnet
-	//td.multiTx1Tr1.Token = &transfer.SntMainnet
-
 	td.multiTx1 = transfer.GenerateTestSendMultiTransaction(td.multiTx1Tr1)
 	td.multiTx1.ToToken = testutils.DaiSymbol
 	td.multiTx1ID = transfer.InsertTestMultiTransaction(t, db, &td.multiTx1)
@@ -302,8 +299,7 @@ func TestGetActivityEntriesWithSameTransactionForSenderAndReceiverInDB(t *testin
 
 	require.Equal(t, SendAT, entries[0].activityType)
 	require.NotEqual(t, eth.Address{}, entries[0].transaction.Address)
-	// TODO: extract and use to/from Address to compare instead of identity
+	require.Equal(t, td.tr1.To, *entries[0].recipient)
-	require.Equal(t, td.tr1.To, entries[0].transaction.Address)
 
 	entries, err = getActivityEntries(context.Background(), deps, []eth.Address{}, []common.ChainID{}, filter, 0, 10)
 	require.NoError(t, err)
@@ -909,15 +905,45 @@ func TestGetActivityEntriesFilterByToAddresses(t *testing.T) {
 	require.NoError(t, err)
 	require.Equal(t, 2, len(entries))
 }
+
 func TestGetActivityEntriesFilterByNetworks(t *testing.T) {
 	deps, close := setupTestActivityDB(t)
 	defer close()
 
 	// Adds 4 extractable transactions
 	td, fromTds, toTds := fillTestData(t, deps.db)
+
+	chainToEntryCount := make(map[common.ChainID]map[int]int)
+	recordPresence := func(chainID common.ChainID, entry int) {
+		if _, ok := chainToEntryCount[chainID]; !ok {
+			chainToEntryCount[chainID] = make(map[int]int)
+			chainToEntryCount[chainID][entry] = 1
+		} else {
+			if _, ok := chainToEntryCount[chainID][entry]; !ok {
+				chainToEntryCount[chainID][entry] = 1
+			} else {
+				chainToEntryCount[chainID][entry]++
+			}
+		}
+	}
+	recordPresence(td.tr1.ChainID, 0)
+	recordPresence(td.pendingTr.ChainID, 1)
+	recordPresence(td.multiTx1Tr1.ChainID, 2)
+	if td.multiTx1Tr2.ChainID != td.multiTx1Tr1.ChainID {
+		recordPresence(td.multiTx1Tr2.ChainID, 2)
+	}
+	recordPresence(td.multiTx2Tr1.ChainID, 3)
+	if td.multiTx2Tr2.ChainID != td.multiTx2Tr1.ChainID {
+		recordPresence(td.multiTx2Tr2.ChainID, 3)
+	}
+	if td.multiTx2PendingTr.ChainID != td.multiTx2Tr1.ChainID && td.multiTx2PendingTr.ChainID != td.multiTx2Tr2.ChainID {
+		recordPresence(td.multiTx2PendingTr.ChainID, 3)
+	}
+
 	// Add 6 extractable transactions
 	trs, fromTrs, toTrs := transfer.GenerateTestTransfers(t, deps.db, td.nextIndex, 6)
 	for i := range trs {
+		recordPresence(trs[i].ChainID, 4+i)
 		transfer.InsertTestTransfer(t, deps.db, trs[i].To, &trs[i])
 	}
 	mockTestAccountsWithAddresses(t, deps.db, append(append(append(fromTds, toTds...), fromTrs...), toTrs...))
@@ -931,14 +957,67 @@ func TestGetActivityEntriesFilterByNetworks(t *testing.T) {
 	chainIDs = []common.ChainID{5674839210}
 	entries, err = getActivityEntries(context.Background(), deps, []eth.Address{}, chainIDs, filter, 0, 15)
 	require.NoError(t, err)
-	// TODO: update after multi-transactions are filterable by ChainID
+	require.Equal(t, 0, len(entries))
-	require.Equal(t, 2 /*0*/, len(entries))
 
 	chainIDs = []common.ChainID{td.pendingTr.ChainID, td.multiTx2Tr1.ChainID, trs[3].ChainID}
 	entries, err = getActivityEntries(context.Background(), deps, []eth.Address{}, chainIDs, filter, 0, 15)
 	require.NoError(t, err)
-	// TODO: update after multi-transactions are filterable by ChainID
+	expectedResults := make(map[int]int)
-	require.Equal(t, 8 /*6*/, len(entries))
+	for _, chainID := range chainIDs {
+		for entry := range chainToEntryCount[chainID] {
+			if _, ok := expectedResults[entry]; !ok {
+				expectedResults[entry]++
+			}
+		}
+	}
+	require.Equal(t, len(expectedResults), len(entries))
+}
+
+func TestGetActivityEntriesFilterByNetworksOfSubTransactions(t *testing.T) {
+	deps, close := setupTestActivityDB(t)
+	defer close()
+
+	// Add 6 extractable transactions
+	trs, _, toTrs := transfer.GenerateTestTransfers(t, deps.db, 0, 5)
+	trs[0].ChainID = 1231
+	trs[1].ChainID = 1232
+	trs[2].ChainID = 1233
+	mt1 := transfer.GenerateTestBridgeMultiTransaction(trs[0], trs[1])
+	trs[0].MultiTransactionID = transfer.InsertTestMultiTransaction(t, deps.db, &mt1)
+	trs[1].MultiTransactionID = mt1.MultiTransactionID
+	trs[2].MultiTransactionID = mt1.MultiTransactionID
+
+	trs[3].ChainID = 1234
+	mt2 := transfer.GenerateTestSwapMultiTransaction(trs[3], testutils.SntSymbol, 100)
+	trs[3].MultiTransactionID = transfer.InsertTestMultiTransaction(t, deps.db, &mt2)
+
+	for i := range trs {
+		if i == 2 {
+			transfer.InsertTestPendingTransaction(t, deps.db, &trs[i])
+		} else {
+			transfer.InsertTestTransfer(t, deps.db, trs[i].To, &trs[i])
+		}
+	}
+
+	var filter Filter
+	chainIDs := allNetworksFilter()
+	entries, err := getActivityEntries(context.Background(), deps, toTrs, chainIDs, filter, 0, 15)
+	require.NoError(t, err)
+	require.Equal(t, 3, len(entries))
+
+	// Extract sub-transactions by pending
+	chainIDs = []common.ChainID{trs[0].ChainID, trs[1].ChainID}
+	entries, err = getActivityEntries(context.Background(), deps, toTrs, chainIDs, filter, 0, 15)
+	require.NoError(t, err)
+	require.Equal(t, 1, len(entries))
+	require.Equal(t, entries[0].id, mt1.MultiTransactionID)
+
+	// Extract sub-transactions by
+	chainIDs = []common.ChainID{trs[2].ChainID}
+	entries, err = getActivityEntries(context.Background(), deps, toTrs, chainIDs, filter, 0, 15)
+	require.NoError(t, err)
+	require.Equal(t, 1, len(entries))
+	require.Equal(t, entries[0].id, mt1.MultiTransactionID)
 }
 
 func TestGetActivityEntriesCheckToAndFrom(t *testing.T) {
@@ -963,8 +1042,7 @@ func TestGetActivityEntriesCheckToAndFrom(t *testing.T) {
 
 	require.Equal(t, SendAT, entries[5].activityType)                  // td.tr1
 	require.NotEqual(t, eth.Address{}, entries[5].transaction.Address) // td.tr1
-	// TODO: extract to/from Address and use it for comparison instead of identity address
+	require.Equal(t, td.tr1.To, *entries[5].recipient)                 // td.tr1
-	require.Equal(t, td.tr1.To, entries[5].transaction.Address) // td.tr1
 
 	require.Equal(t, SendAT, entries[4].activityType) // td.pendingTr
 
@@ -982,8 +1060,6 @@ func TestGetActivityEntriesCheckToAndFrom(t *testing.T) {
 	// TODO: Test with all addresses
 }
 
-// TODO test sub-transaction count for multi-transactions
-
 func TestGetActivityEntriesCheckContextCancellation(t *testing.T) {
 	deps, close := setupTestActivityDB(t)
 	defer close()