Improve the error management in order to avoid DB corruption in case the process is killed while encrypting the DB.
Changes:
Use sqlcipher_export instead of rekey to change the DB password. The advantage is that sqlcipher_export will operate on a new DB file and we don't need to modify the current account unless the export is successful.
Keeping the rekey requires to create a DB copy before trying to re-encrypt the DB, but the DB copy is risky in case the DB file changes wile the copy is in progress. It could also lead to DB corruption.
* perf(sqlCipher): Increase cipher page size to 8192
Increasing the cipher page size to 8192 requires DB re-encryption. The process is as follows:
//Login to v3 DB
PRAGMA key = 'key';
PRAGMA cipher_page_size = 1024"; // old Page size
PRAGMA cipher_hmac_algorithm = HMAC_SHA1";
PRAGMA cipher_kdf_algorithm = PBKDF2_HMAC_SHA1";
PRAGMA kdf_iter = kdfIterationsNumber";
//Create V4 DB with increased page size
ATTACH DATABASE 'newdb.db' AS newdb KEY 'key';
PRAGMA newdb.cipher_page_size = 8192; // new Page size
PRAGMA newdb.cipher_hmac_algorithm = HMAC_SHA1"; // same as in v3
PRAGMA newdb.cipher_kdf_algorithm = PBKDF2_HMAC_SHA1"; // same as in v3
PRAGMA newdb.kdf_iter = kdfIterationsNumber"; // same as in v3
SELECT sqlcipher_export('newdb');
DETACH DATABASE newdb;
//Login to V4 DB
...
Worth noting:
The DB migration will happen on the first successful login.
The new DB version will have a different name to be able to distinguish between different DB versions.Versions naming mirrors sqlcipher major version (naming conventions used by sqlcipher), meaning that we're migrating from V3 to V4 DB (even if we're not fully aligned with V4 standards). The DB is not migrated to the v4 standard `SHA512` due to performance reasons. Our custom `SHA1` implementation is fully optimised for perfomance.
* perf(sqlCipher): Fixing failing tests
Update the new DB file format in Delete account, Change password and Decrypt database flows
* perf(SQLCipher): Increase page size - send events to notify when the DB re-encryption starts/ends
Extended the migration process with a generic way of applying custom
migration code on top of the SQL files. The implementation provides
a safer way to run GO code along with the SQL migrations and possibility
of rolling back the changes in case of failure to keep the database
consistent.
This custom GO migration is needed to extract the status from
the JSON blob receipt and store it in transfers table.
Other changes:
- Add NULL DB value tracking to JSONBlob helper
- Index status column on transfers table
- Remove unnecessary panic calls
- Move log_parser to wallet's common package and use to extract token
identity from the logs
Notes:
- there is already an index on transfers table, sqlite creates one for
each unique constraint therefore add only status to a new index
- the planned refactoring and improvements to the database have been
postponed due to time constraints. Got the time to migrate the data
though, extracting it can be done later for a more efficient
implementation
Update status-desktop #10746
* chore(upgradeSQLCipher): Upgrading SQLCipher to version 5.4.5
Changes:
### github.com/mutecomm/go-sqlcipher
1. The improved crypto argorighms from go-sqlcipher v3 are merged in v4
Tags:
v4.4.2-status.1 - merge `burn_stack` improvement
v4.4.2-status.2 - merge `SHA1` improvement
v4.4.2-status.4- merge 'AES' improvement
2. Fixed `go-sqlcipher` to support v3 database in compatibility mode (`sqlcipher` already supports this) (Tag: v4.4.2-status.3)
3. Upgrade `sqlcipher` to v5.4.5 (Tag: v4.5.4-status.1)
### github.com/status-im/migrate/v4
1. Upgrade `go-sqlcipher` version in `github.com/status-im/migrate/v4`
### status-go
1. Upgrade `go-sqlcipher` and `migrate` modules in status-go
2. Configure the DB connections to open the DB in v3 compatibility mode
* chore(upgradeSQLCipher): Use sqlcipher v3 configuration to encrypt a plain text database
* chore(upgradeSQLCipher): Scanning NULL BLOB value should return nil
Fixing failing tests: TestSyncDeviceSuite/TestPairingSyncDeviceClientAsReceiver; TestSyncDeviceSuite/TestPairingSyncDeviceClientAsSender
Considering the following configuration:
1. Table with BLOB column has 1 NULL value
2. Query the value
3. Rows.Scan(&dest sql.NullString)
Expected: dest.Valid == false; dest.String == nil
Actual: dest.Valid == true; dest.String == ""
* chore: Bump go-sqlcipher version to include NULL BLOB fix
The default transaction lock is `deferred`. This means that the transaction will automatically become read or write transaction based on the first DB operation. In case the first operation is `SELECT` the transaction becomes read transaction, otherwise write transaction. When a read transaction tries to write the DB sqlite will promote the transaction to a write transaction if there is no other transaction that holds a lock. When the promotion fails `database is locked` error is returned. The error is returned immediately and does not use the busy handler.
In our case almost all read transaction would fail with `database is locked` error.
This fix is changing the default transaction lock to `IMMEDIATE`. It translates to `BEGIN IMMEDIATE` instead of `BEGIN`. In this mode, the transaction will be created as a write transaction no matter what DB operation will run as part of the transaction. The write transaction will try to obtain the DB lock immediately when `BEGIN IMMEDIATE` is called and the busy handler is used when the DB is locked by other transaction.
Fixing: https://github.com/status-im/status-desktop/issues/10838
This commit does a few things:
1) Extend create/import account endpoint to get wallet config, some of
which has been moved to the backend
2) Set up a loop for retrieving balances every 10 minutes, caching the
balances
3) Return information about which checks are not passing when trying to
join a token gated community
4) Add tests to the token gated communities
5) Fixes an issue with addresses not matching when checking for
permissions
The move to the wallet as a background task is not yet complete, I need
to publish a signal, and most likely I will disable it before merging
for now, as it's currently not used by desktop/mobile, but the PR was
getting to big
prefixes. Changed primary keys and API methods.
Fixed tests and added new ones.
Fixed saved addresses and transaction tests to use ':memory:' sqlite
DB instead of a tmp file to speed up testing by hundred of times.
Fixes#8599
* Sync Settings
* Added valueHandlers and Database singleton
Some issues remain, need a way to comparing incoming sql.DB to check if the connection is to a different file or not. Maybe make singleton instance per filename
* Added functionality to check the sqlite filename
* Refactor of Database.SaveSyncSettings to be used as a handler
* Implemented inteface for setting sync protobuf factories
* Refactored and completed adhoc send setting sync
* Tidying up
* Immutability refactor
* Refactor settings into dedicated package
* Breakout structs
* Tidy up
* Refactor of bulk settings sync
* Bug fixes
* Addressing feedback
* Fix code dropped during rebase
* Fix for db closed
* Fix for node config related crashes
* Provisional fix for type assertion - issue 2
* Adding robust type assertion checks
* Partial fix for null literal db storage and json encoding
* Fix for passively handling nil sql.DB, and checking if elem has len and if len is 0
* Added test for preferred name behaviour
* Adding saved sync settings to MessengerResponse
* Completed granular initial sync and clock from network on save
* add Settings to isEmpty
* Refactor of protobufs, partially done
* Added syncSetting receiver handling, some bug fixes
* Fix for sticker packs
* Implement inactive flag on sync protobuf factory
* Refactor of types and structs
* Added SettingField.CanSync functionality
* Addressing rebase artifact
* Refactor of Setting SELECT queries
* Refactor of string return queries
* VERSION bump and migration index bump
* Deactiveate Sync Settings
* Deactiveated preferred_name and send_status_updates
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit introduces the following changes:
- `local-notifications` require as body an interface complying with
`json.Marshaler`
- removed unmarshaling of `Notifications` as not used (we only Marshal
notifications)
- `protocol/messenger.go` creates directly a `Notification` instead of
having an intermediate format
- add community notifications on request to join
- move parsing of text in status-go for notifications
* WIP accounts implementation
* Accounts datasore and changes to status mobile API
* Add library changes and method to update config
* Handle error after account selection
* Add two methods to start account to backend
* Use encrypted database for settings and add a service for them
* Resolve linter warning
* Bring back StartNode StopNode for tests
* Add sub accounts and get/save api
* Changes to accounts structure
* Login use root address and fetch necessary info from database
* Cover accounts store with tests
* Refactor in progress
* Initialize status keystore instance before starting ethereum node
* Rework library tests
* Resolve failures in private api test and send transaction test
* Pass pointer to initialized config to unmarshal
* Use multiaccounts/accounts naming consistently
Multiaccount is used as a login identifier
Account references an address and a key, if account is not watch-only.
* Add login timestamp stored in the database to accounts.Account object
* Add photo-path field for multiaccount struct
* Add multiaccoutns rpc with updateAccount method
Update to any other account that wasn't used for login will return an error
* Fix linter in services/accounts
* Select account before starting a node
* Save list of accounts on first login
* Pass account manager to accounts service to avoid selecting account before starting a node
* Add logs to login with save and regualr login
Dario Gabriel Lipicar
Alex Jbanca
Stefan
Alex Jbanca
Patryk Osmaczko
Andrea Maria Piana
IvanBelyakoff
Richard Ramos
Samuel Hawksby-Robinson
Shivek Khurana
Dmitry Shulyak