163 lines
5.7 KiB
Markdown
163 lines
5.7 KiB
Markdown
# DB extension and refactoring for activity
|
|
|
|
## Work in progress
|
|
|
|
| | Buy | Swap | Bridge | Send/Receive |
|
|
| ------------- | -------------- | -------------- | ------------- | ------------- |
|
|
| Activity data | ~~API~~ ~~DB~~ | ~~API~~ DB | _API_ DB | _API_ DB |
|
|
| Raw data | ~~API~~ ~~DB~~ | ~~API~~ DB | _API_ DB | _API_ DB |
|
|
| Pending data | ~~API~~ ~~DB~~ | ~~API~~ DB | _API_ DB | _API_ DB |
|
|
|
|
Legend:
|
|
|
|
- ~~API~~ - not much or at all provided
|
|
- _API_ - partially provided
|
|
- API - complete
|
|
|
|
### Summary
|
|
|
|
Improve on the identified limitations
|
|
|
|
- [ ] Missing filtering data
|
|
- [x] Missing cached (not extracted as a column)
|
|
- Extracting the data from the raw data is expensive but might be negligible given that usually we should not expect more than 20 entries per second in the worst case scenario.
|
|
- [x] Table extensions
|
|
- ~~Activity specific info in activity data store (multi_transaction table)~~
|
|
- Activity specific info in in the transactions data store (transfers table)
|
|
|
|
### Missing data
|
|
|
|
Filter requirements
|
|
|
|
- [x] Activity operation status
|
|
- [x] `pending`: have to aggregate for `Buy`, `Swap`, `Bridge`
|
|
- already there for `Send`, `Receive`
|
|
- [x] `complete`: only extract and check for `status` in the `receipt` for `Send`, `Receive`
|
|
- For complex operations aggregate the `complete` status `Buy`, `Swap`, `Bridge`
|
|
- [x] `finalized`: similar to `complete` for `Send`, `Receive`
|
|
- all sub-transactions are `complete` for `Buy`, `Swap`, `Bridge`
|
|
- [x] `failed`: extract from `status` for all sub-transactions
|
|
- [x] `chainID`: aggregate data for activity entries `Bridge`, `Buy`, `Swap`
|
|
- [x] `tokenCode` for activity entries `Send`, `Receive`
|
|
- For `Bridge` its already there and `Buy`, `Swap` is coming soon
|
|
- [ ] `collectibles`: require adding collectible attributes to activity data (`token_address` and `tokenId`)
|
|
|
|
UX requirements
|
|
|
|
- [x] `status`: for status icon and label
|
|
- [ ] `chainIDs`: for chain icons
|
|
- Missing for `Bridge`, `Buy`, `Swap`
|
|
- [ ] `amount`s: add to the activity.Entry
|
|
- already in DB
|
|
- [ ] `tokenCode`s: add to the activity.Entry
|
|
- already in DB
|
|
- [ ] `to`/`from`/`owner`: add to the activity.Entry
|
|
- already in DB, coming soon
|
|
- [ ] `tokenIdentity`: collectible is missing (`chainId`, `address`, `tokenId`)
|
|
- `tokenCode` should be covering fungible operations
|
|
- [x] `identity`: for all the sources
|
|
- [x] `type`: for the main icon and label
|
|
- [x] `time`: timestamp
|
|
|
|
### Refactoring
|
|
|
|
Extend `entry.nim:ActivityEntry` and `activity.go:Entry` with presentation layer data
|
|
|
|
- [ ] `activityType`: instead of the current `MultiTransactionType`
|
|
- [x] `status`: for status icon and label
|
|
|
|
## Current state
|
|
|
|
### Transfers Table
|
|
|
|
The `transfers` transactions raw data
|
|
|
|
- Transaction identity: `network_id`, `hash`, `address`
|
|
- Implementation by `sqlite_autoindex_transfers_1` unique index
|
|
- `multi_transaction_id`: `multi_transaction` entries to `transfers` entries mapping (one to many)
|
|
- Raw data:
|
|
- `tx` transaction
|
|
- `receipt`: transfer receipt
|
|
|
|
### Multi-Transaction Table
|
|
|
|
Represented by `multi_transaction`
|
|
|
|
Responsibilities
|
|
|
|
- UX metadata for transactions originating from our APP.
|
|
- `from_address`, `to_address`
|
|
- `from_asset`, `to_asset`
|
|
- `from_amount`, `to_amount` (token codes)
|
|
- `type` identifies the type (initially only Send and Bridge)
|
|
- `timestamp` the timestamp of the execution
|
|
- Multi-transaction to sub-transaction mapping
|
|
- The `multi_transaction_id` in the `transfers` and `pending_transaction` table corresponds to the `ROWID` in the `multi_transactions`.
|
|
|
|
### Pending Transactions Table
|
|
|
|
The `pending_transactions` table represents transactions initiated from the app
|
|
|
|
- Transaction identity
|
|
- `network_id`, `hash`
|
|
- implemented by the `sqlite_autoindex_pending_transactions_1` index
|
|
- Note how this is different from the `transfers` table, where the `address` is also part of the identity.
|
|
- `timestamp`: The timestamp of the pending transaction.
|
|
- `multi_transaction_id`: `multi_transaction` entries to `pending_transactions` entries mapping (one to many)
|
|
|
|
### Schema
|
|
|
|
Relationships between the tables
|
|
|
|
```mermaid
|
|
erDiagram
|
|
multi_transaction ||--o{ transfers : has
|
|
multi_transaction ||--o{ pending_transactions : has
|
|
transfers {
|
|
network_id BIGINT
|
|
hash VARCHAR
|
|
timestamp BIGINT
|
|
multi_transaction_id INT
|
|
tx BLOB
|
|
receipt BLOB
|
|
log BLOB
|
|
}
|
|
multi_transaction {
|
|
ROWID INT
|
|
type VARCHAR
|
|
}
|
|
pending_transactions {
|
|
network_id BIGINT
|
|
hash VARCHAR
|
|
timestamp INT
|
|
multi_transaction_id INT
|
|
}
|
|
```
|
|
|
|
### Dropped tasks
|
|
|
|
Dropped the DB refactoring and improvements after further discussion and concerns
|
|
|
|
- [x] Terminology proposal
|
|
- [x] using `transactions` instead of `transfers` for the raw data
|
|
- [x] using `activity` instead of `multi-transaction` to better match the new requirements
|
|
- [x] Convert JSON blobs into structured data
|
|
|
|
Dropped benchmark performance and move on using theoretical knowledge by adding indexes for what we know only
|
|
Will leave the performance concerns for the next milestone
|
|
|
|
- [ ] Joining DBs
|
|
- [ ] One activity DB for all require metadata
|
|
- Pros:
|
|
- Faster to query (don't know the numbers)
|
|
- Simpler query will decrease maintenance
|
|
- Cons:
|
|
- have to migrate all data, extract and fill the activity on every download of updates for all activities
|
|
- [ ] Keep only filter specific metadata in the Activity DB
|
|
- Pros:
|
|
- Less changes to migrate existing data. Still have to maintain activity filtering specific data
|
|
- Cons:
|
|
- Slower to query (don't know how much yet)
|
|
- Complex query increases maintenance
|
|
|