status-go/services/wallet/activity/TODO.md

DB extension and refactoring for activity

Work in progress

	Buy	Swap	Bridge	Send/Receive
Activity data	API DB	API DB	API DB	API DB
Raw data	API DB	API DB	API DB	API DB
Pending data	API DB	API DB	API DB	API DB

Legend:

• API - not much or at all provided
• API - partially provided
• API - complete

Summary

Improve on the identified limitations

• Missing filtering data
• Missing cached (not extracted as a column)
  • Extracting the data from the raw data is expensive but might be negligible given that usually we should not expect more than 20 entries per second in the worst case scenario.
• Table extensions
  • Activity specific info in activity data store (multi_transaction table)
  • Activity specific info in in the transactions data store (transfers table)

Missing data

Filter requirements

• Activity operation status
  • pending: have to aggregate for Buy, Swap, Bridge
    • already there for Send, Receive
  • complete: only extract and check for status in the receipt for Send, Receive
    • For complex operations aggregate the complete status Buy, Swap, Bridge
  • finalized: similar to complete for Send, Receive
    • all sub-transactions are complete for Buy, Swap, Bridge
  • failed: extract from status for all sub-transactions
• chainID: aggregate data for activity entries Bridge, Buy, Swap
• tokenCode for activity entries Send, Receive
  • For Bridge its already there and Buy, Swap is coming soon
• collectibles: require adding collectible attributes to activity data (probably token_address and tokenId)

UX requirements

• status: for status icon and label
• chainIDs: for chain icons
  • Missing for Bridge, Buy, Swap
• amounts: add to the activity.Entry
  • already in DB
• tokenCodes: add to the activity.Entry
  • already in DB
• to/from/owner: add to the activity.Entry
  • already in DB, coming soon
• tokenIdentity: collectible is missing (chainId, address, tokenId)
  • tokenCode should be covering fungible operations
• identity: for all the sources
• type: for the main icon and label
• time: timestamp

Refactoring

Extend entry.nim:ActivityEntry and activity.go:Entry with presentation layer data

• activityType: instead of the current MultiTransactionType
• status: for status icon and label

Current state

Transfers Table

The transfers transactions raw data

• Transaction identity: network_id, hash, address
  • Implementation by sqlite_autoindex_transfers_1 unique index
• multi_transaction_id: multi_transaction entries to transfers entries mapping (one to many)
• Raw data:
  • tx transaction
  • receipt: transfer receipt

Multi-Transaction Table

Represented by multi_transaction

Responsibilities

• UX metadata for transactions originating from our APP.
  • from_address, to_address
  • from_asset, to_asset
  • from_amount, to_amount (token codes)
  • type identifies the type (initially only Send and Bridge)
  • timestamp the timestamp of the execution
• Multi-transaction to sub-transaction mapping
  • The multi_transaction_id in the transfers and pending_transaction table corresponds to the ROWID in the multi_transactions.

Pending Transactions Table

The pending_transactions table represents transactions initiated from the app

• Transaction identity
  • network_id, hash
    • implemented by the sqlite_autoindex_pending_transactions_1 index
    • Note how this is different from the transfers table, where the address is also part of the identity.
• timestamp: The timestamp of the pending transaction.
• multi_transaction_id: multi_transaction entries to pending_transactions entries mapping (one to many)

Schema

Relationships between the tables

erDiagram
    multi_transaction ||--o{ transfers : has
    multi_transaction ||--o{ pending_transactions : has
    transfers {
        network_id BIGINT
        hash VARCHAR
        timestamp BIGINT
        multi_transaction_id INT
        tx BLOB
        receipt BLOB
        log BLOB
    }
    multi_transaction {
      ROWID INT
      type VARCHAR
    }
    pending_transactions {
        network_id BIGINT
        hash VARCHAR
        timestamp INT
        multi_transaction_id INT
    }

Dropped tasks

Dropped the DB refactoring and improvements after further discussion and concerns

• Terminology proposal
  • using transactions instead of transfers for the raw data
  • using activity instead of multi-transaction to better match the new requirements
  • Convert JSON blobs into structured data

Dropped benchmark performance and move on using theoretical knowledge by adding indexes for what we know only Will leave the performance concerns for the next milestone

• Joining DBs
• One activity DB for all require metadata
  • Pros:
    • Faster to query (don't know the numbers)
    • Simpler query will decrease maintenance
  • Cons:
    • have to migrate all data, extract and fill the activity on every download of updates for all activities
• Keep only filter specific metadata in the Activity DB
  • Pros:
    • Less changes to migrate existing data. Still have to maintain activity filtering specific data
  • Cons:
    • Slower to query (don't know how much yet)
    • Complex query increases maintenance