Extended the migration process with a generic way of applying custom
migration code on top of the SQL files. The implementation provides
a safer way to run GO code along with the SQL migrations and possibility
of rolling back the changes in case of failure to keep the database
consistent.
This custom GO migration is needed to extract the status from
the JSON blob receipt and store it in transfers table.
Other changes:
- Add NULL DB value tracking to JSONBlob helper
- Index status column on transfers table
- Remove unnecessary panic calls
- Move log_parser to wallet's common package and use to extract token
identity from the logs
Notes:
- there is already an index on transfers table, sqlite creates one for
each unique constraint therefore add only status to a new index
- the planned refactoring and improvements to the database have been
postponed due to time constraints. Got the time to migrate the data
though, extracting it can be done later for a more efficient
implementation
Update status-desktop #10746
* chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5
Changes:
### github.com/mutecomm/go-sqlcipher
1. The improved crypto argorighms from go-sqlcipher v3 are merged in v4
Tags:
v4.4.2-status.1 - merge `burn_stack` improvement
v4.4.2-status.2 - merge `SHA1` improvement
v4.4.2-status.4- merge 'AES' improvement
2. Fixed `go-sqlcipher` to support v3 database in compatibility mode (`sqlcipher` already supports this) (Tag: v4.4.2-status.3)
3. Upgrade `sqlcipher` to v5.4.5 (Tag: v4.5.4-status.1)
### github.com/status-im/migrate/v4
1. Upgrade `go-sqlcipher` version in `github.com/status-im/migrate/v4`
### status-go
1. Upgrade `go-sqlcipher` and `migrate` modules in status-go
2. Configure the DB connections to open the DB in v3 compatibility mode
* chore(upgradeSQLCipher): Use sqlcipher v3 configuration to encrypt a plain text database
* chore(upgradeSQLCipher): Scanning NULL BLOB value should return nil
Fixing failing tests: TestSyncDeviceSuite/TestPairingSyncDeviceClientAsReceiver; TestSyncDeviceSuite/TestPairingSyncDeviceClientAsSender
Considering the following configuration:
1. Table with BLOB column has 1 NULL value
2. Query the value
3. Rows.Scan(&dest sql.NullString)
Expected: dest.Valid == false; dest.String == nil
Actual: dest.Valid == true; dest.String == ""
* chore: Bump go-sqlcipher version to include NULL BLOB fix
Improve performance of queries for
- transfer.GetTransfersInRange
- transfer.GetTransfersByAddress
- transfer.GetTransfersByAddressAndBlock
- transfer.GetTransfers
- transfer.GetPreloadedTransactions
For 16952 entries worst case scenario tested with `sqlcipher`:
- Before: Run Time: real 0.897 user 0.728139 sys 0.166714
- After: Run Time: real 0.001 user 0.000437 sys 0.000189
A single composite index (with the default one) might work better though
Add support for unfurling a wider range of websites. Most code changes are
related to the implementation of a new Unfurler, an OEmbedUnfurler, which is
necessary to get metadata for Reddit URLs using oEmbed, since Reddit does not
support OpenGraph meta tags. The new unfurler will also be useful for other
websites, like Twitter. Also the user agent was changed, and now more websites
consider status-go reasonably human.
Related to issue https://github.com/status-im/status-mobile/issues/15918
Example hostnames that are now unfurleable: reddit.com, open.spotify.com,
music.youtube.com
Other improvements:
- Better error handling, especially because I wasn't wrapping errors correctly.
I also removed the unnecessary custom error UnfurlErr.
- I made tests truly deterministic by parameterizing the http.Client instance
and by customizing its Transport field (except for some failing conditions
where it's even good to hit the real servers).
The test running time can be a bit unpredictable due to server load,
which can result in tests that finish in 5 minutes, or over 20.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
- reverted a change that stopped looking for ERC20 transfers if no nonce
and balance change found within a block range for ETH
- implemented sending EventRecentHistoryReady event at a proper time
- moved EventFetchingRecentHistory event to Strategy type as it does not make
sense to send this event in loop
- moved iterating through blocks logic to inside of `loadTransfers` command, which
now accepts a block range.
- reuse `uniqueHeaders` function in commands.go
- clean up
Updates #10246
This commit adds LoginAccount endpoint.
This makes it consistent with CreateAccount and RestoreAccount as they
use similar config.
The notable difference with the previous endpoint is the API, which is
the same as CreateAccount/RestoreAccount, and the fact that it will
override your networks configuration.
Storing them in the config is now not needed anymore, as that's always
driven from the backend, and we won't allow custom networks in the new
wallet.
The default transaction lock is `deferred`. This means that the transaction will automatically become read or write transaction based on the first DB operation. In case the first operation is `SELECT` the transaction becomes read transaction, otherwise write transaction. When a read transaction tries to write the DB sqlite will promote the transaction to a write transaction if there is no other transaction that holds a lock. When the promotion fails `database is locked` error is returned. The error is returned immediately and does not use the busy handler.
In our case almost all read transaction would fail with `database is locked` error.
This fix is changing the default transaction lock to `IMMEDIATE`. It translates to `BEGIN IMMEDIATE` instead of `BEGIN`. In this mode, the transaction will be created as a write transaction no matter what DB operation will run as part of the transaction. The write transaction will try to obtain the DB lock immediately when `BEGIN IMMEDIATE` is called and the busy handler is used when the DB is locked by other transaction.
Fixing: https://github.com/status-im/status-desktop/issues/10838
Fixes an issue where if a group chat was first received from a non-contact, and later received from a contact, it still wouldn't save it as active.
That's because we checked if we were **newly** added instead of just if we were added. That meant that in the case I described above, the chat would then never have the chance to be set active.
* fix(community): stop re-joining comm when receiving a sync community msg
Fixes an issue with chats being reset. Since joining a community resaves the chats with the synced default value, it resets the sate of the chats, losing the unread messages, the muted state and more.
The solution is to block the re-joining of the community. In the case of the sync, we catch that error and just continue on.
* fix(import): fix HandleImport not saving the chat
Doesn't change much, but it could have caused issues in the future, so since we might have modified the chat, we make sure to save them
Also adds a test
* fix tests
