mirror of https://github.com/status-im/status-go.git synced 2025-02-18 09:47:53 +00:00

Stefan 5b6f7226bb feat(wallet) extract json blobs and add custom migration support

Extended the migration process with a generic way of applying custom
migration code on top of the SQL files. The implementation provides
a safer way to run GO code along with the SQL migrations and possibility
of rolling back the changes in case of failure to keep the database
consistent.
This custom GO migration is needed to extract the status from
the JSON blob receipt and store it in transfers table.

Other changes:
- Add NULL DB value tracking to JSONBlob helper
- Index status column on transfers table
- Remove unnecessary panic calls
- Move log_parser to wallet's common package and use to extract token
  identity from the logs

Notes:
- there is already an index on transfers table, sqlite creates one for
  each unique constraint therefore add only status to a new index
- the planned refactoring and improvements to the database have been
  postponed due to time constraints. Got the time to migrate the data
  though, extracting it can be done later for a more efficient
  implementation

Update status-desktop #10746

2023-06-08 13:54:40 +02:00

5.5 KiB

Raw Blame History

DB extension and refactoring for activity

Work in progress

	Buy	Swap	Bridge	Send/Receive
Activity data	~~API~~ DB	~~API~~ DB	API DB	API DB
Raw data	~~API~~ DB	~~API~~ DB	API DB	API DB
Pending data	~~API~~ DB	~~API~~ DB	API DB	API DB

Legend:

~~API~~ - not much or at all provided
API - partially provided
API - complete

Summary

Improve on the identified limitations

Missing filtering data
Missing cached (not extracted as a column)
- Extracting the data from the raw data is expensive but might be negligible given that usually we should not expect more than 20 entries per second in the worst case scenario.
Table extensions
- ~~Activity specific info in activity data store (multi_transaction table)~~
- Activity specific info in in the transactions data store (transfers table)

Missing data

Filter requirements

Activity operation status
- pending: have to aggregate for Buy, Swap, Bridge
  - already there for Send, Receive
- complete: only extract and check for status in the receipt for Send, Receive
  - For complex operations aggregate the complete status Buy, Swap, Bridge
- finalized: similar to complete for Send, Receive
  - all sub-transactions are complete for Buy, Swap, Bridge
- failed: extract from status for all sub-transactions
chainID: aggregate data for activity entries Bridge, Buy, Swap
tokenCode for activity entries Send, Receive
- For Bridge its already there and Buy, Swap is coming soon
collectibles: require adding collectible attributes to activity data (probably token_address and tokenId)

UX requirements

status: for status icon and label
chainIDs: for chain icons
- Missing for Bridge, Buy, Swap
amounts: add to the activity.Entry
- already in DB
tokenCodes: add to the activity.Entry
- already in DB
to/from/owner: add to the activity.Entry
- already in DB, coming soon
tokenIdentity: collectible is missing (chainId, address, tokenId)
- tokenCode should be covering fungible operations
identity: for all the sources
type: for the main icon and label
time: timestamp

Refactoring

Current state

Transfers Table

The transfers transactions raw data

Transaction identity: network_id, hash, address
- Implementation by sqlite_autoindex_transfers_1 unique index
multi_transaction_id: multi_transaction entries to transfers entries mapping (one to many)
Raw data:
- tx transaction
- receipt: transfer receipt

Multi-Transaction Table

Represented by multi_transaction

Responsibilities

UX metadata for transactions originating from our APP.
- from_address, to_address
- from_asset, to_asset
- from_amount, to_amount (token codes)
- type identifies the type (initially only Send and Bridge)
- timestamp the timestamp of the execution
Multi-transaction to sub-transaction mapping
- The multi_transaction_id in the transfers and pending_transaction table corresponds to the ROWID in the multi_transactions.

Pending Transactions Table

The pending_transactions table represents transactions initiated from the app

Transaction identity
- network_id, hash
  - implemented by the sqlite_autoindex_pending_transactions_1 index
  - Note how this is different from the transfers table, where the address is also part of the identity.
timestamp: The timestamp of the pending transaction.
multi_transaction_id: multi_transaction entries to pending_transactions entries mapping (one to many)

Schema

Relationships between the tables

erDiagram
    multi_transaction ||--o{ transfers : has
    multi_transaction ||--o{ pending_transactions : has
    transfers {
        network_id BIGINT
        hash VARCHAR
        timestamp BIGINT
        multi_transaction_id INT
        tx BLOB
        receipt BLOB
        log BLOB
    }
    multi_transaction {
      ROWID INT
      type VARCHAR
    }
    pending_transactions {
        network_id BIGINT
        hash VARCHAR
        timestamp INT
        multi_transaction_id INT
    }

Dropped tasks

Dropped the DB refactoring and improvements after further discussion and concerns

Terminology proposal
- using transactions instead of transfers for the raw data
- using activity instead of multi-transaction to better match the new requirements
- Convert JSON blobs into structured data

Dropped benchmark performance and move on using theoretical knowledge by adding indexes for what we know only Will leave the performance concerns for the next milestone

Joining DBs
One activity DB for all require metadata
- Pros:
  - Faster to query (don't know the numbers)
  - Simpler query will decrease maintenance
- Cons:
  - have to migrate all data, extract and fill the activity on every download of updates for all activities
Keep only filter specific metadata in the Activity DB
- Pros:
  - Less changes to migrate existing data. Still have to maintain activity filtering specific data
- Cons:
  - Slower to query (don't know how much yet)
  - Complex query increases maintenance

5.5 KiB Raw Blame History