mirror of https://github.com/waku-org/nwaku.git
DB Migration tutorial (#646)
* adds timestamp to waku message store impl * stores timestamp as int64 * adds untitest * stores timestamp as seq of bytes * minor * re-orders unittest * changes receiver timestamp to float64 * unit test for receiver timestamps * adds comments * reorder a few lines * updates changelog * more updates on changelog * WIP * WIP * adds migration * more debug messages * passes the path to the migration scripts from message store module * adds migration result type * replaces migrationScripts with migrationScriptsResult * adds path calculation to the message store * removes some tests binary file * removes redundant imports * comments out user_version assignment in sqlite init * more descriptive err messages * clean up test file * more info logs * minor code format * removes a todo * minor updates * remove a binary file * unit tests for migration utils * adds split script * integrates split query to handle scripts with multiple commands * updates migration script for v1 * updates the v1 migration script * update user version * updates script * fixes a few bugs on the splitScript * more debug logs * adds float64 parameter support to sqlite3 * change in timestamp type in the script * deletes float64 toBytes utils * enables storage of timestamp as a real number in the sqlite db * bump up script index * comment edits * removes migrate unit test * adds todo and docstring * updates changelog * removes an unused item in .gitignore * minor * updates changelog * organizes imports * cleans up imports * WIP * updates script fixes a few bugs on the splitScript more debug logs adds float64 parameter support to sqlite3 change in timestamp type in the script deletes float64 toBytes utils * edits migration util test * remove an empty test file * includes migration utils tests in * deletes unused codes * tides up imports * adds range based filter to the filterMigrationScripts * renames procs: removes Migration * tides up imports * edits docstring * edits docstring * edits docstring * removes unused imports * more clean up * groups std imports * updates changelog * adds docstring for setUserVersion * adds unittest for the migrate * Update waku/v2/node/storage/message/waku_message_store.nim Co-authored-by: RichΛrd <info@richardramos.me> * Update waku/v2/node/storage/sqlite.nim Co-authored-by: RichΛrd <info@richardramos.me> * Update waku/v2/node/storage/sqlite.nim Co-authored-by: RichΛrd <info@richardramos.me> * removes split scripts * fixes a naming issue * fixes a bug * fixes a typo * adds a log re updated user_version * fixes a proc naming mismatch * fixes naming mismatch * more descriptive var names * adds migration script of the first user version * moves migration to after persistMessages flag is checked * deletes unused comment * fixes a bug * brings back split script * adds unit tests for split scripts * runs scripts one command at a time * deletes a commented line * relocates the migrate proc to sqlite.nim * initial writeup * adds migration tutorial * adds User guides to db migration tutorial * minor * proofread * replaces a broken link * user-version to user_version * some clarification * some edits * minor * a quick wording fix * minor * final edits * updates the changelog * slight change Co-authored-by: RichΛrd <info@richardramos.me>
This commit is contained in:
Sanaz Taheri Boshrooyeh
GitHub
parent
d4974049d9
commit
05cb073a0f
|
|
@ -11,6 +11,7 @@ This release contains the following:
|
|||
|
||||
#### General refactoring
|
||||
#### Docs
|
||||
- Adds the database migration tutorial.
|
||||
#### Schema
|
||||
- Updates the `Message` table of the persistent message store:
|
||||
- Adds `senderTimestamp` column.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,62 @@
|
|||
# Database Migration
|
||||
This tutorial explains the database migration process in nim-waku.
|
||||
|
||||
# Contributors Guide
|
||||
## Database Migration Flow
|
||||
Nim-waku utilizes the built-in `user_version` variable that Sqlite provides for tracking the database versions.
|
||||
The [user_version](https://github.com/status-im/nim-waku/blob/master/waku/v2/node/storage/migration/migration_types.nim) MUST be bumped up for every update on the database e.g, table schema/title change.
|
||||
Each update should be accompanied by a migration script to move the content of the old version of the database to the new version.
|
||||
The script MUST be added to the respective folder as explained in [Migration Folder Structure](#migration-folder-structure) with the proper naming as given in [ Migration Script Naming](#migration-file-naming).
|
||||
|
||||
The migration is invoked whenever the database `user_version` is behind the target [user_version](https://github.com/status-im/nim-waku/blob/master/waku/v2/node/storage/migration/migration_types.nim) indicated in the nim-waku application.
|
||||
The respective migration scripts located in the [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) will be executed to upgrade the database from its old version to the target version.
|
||||
|
||||
## Migration Folder Structure
|
||||
The [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) is structured as below.
|
||||
|
||||
```
|
||||
|-- migration
|
||||
| |--migration_scripts
|
||||
| | |--message
|
||||
| | | |--00001_basicMessageTable.up.sql
|
||||
| | | |--00002_addSenderTimeStamp.up
|
||||
| | | |-- ...
|
||||
| | |--peer
|
||||
| | | |--00001_basicPeerTable.up.sql
|
||||
| | | |-- ...
|
||||
```
|
||||
|
||||
The migration scripts are managed in two separate folders `message` and `peer`.
|
||||
The `message` folder contains the migration scripts related to the message store tables.
|
||||
Similarly, the `peer` folder contains the scripts relevant to the peer store tables.
|
||||
|
||||
|
||||
## Migration File Naming
|
||||
The files in [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) MUST follow the following naming style in order to be properly included in the migration process.
|
||||
Files with invalid naming will be eliminated from the migration process.
|
||||
|
||||
`<version number>_<migration script description>.<up|down>.sql`
|
||||
|
||||
- `version number`: This number should match the target value of `user_version`.
|
||||
- `migration script description`: A short description of what the migration script does.
|
||||
- `up|down`: One of the keywords of `up` or `down` should be selected.
|
||||
`up` stands for upgrade and `down` means downgrade.
|
||||
|
||||
### Example
|
||||
A migration file with the name `00002_addTableX.up.sql` should be interpreted as:
|
||||
- `00002`: The targeted `user_version` number.
|
||||
- `addTableX`: What the script does.
|
||||
- `up`: This script `upgrade`s the database from `user_version = 00001` to the `user_version = 00002`.
|
||||
|
||||
A downgrade migration file corresponding to the `00002_addTableX.up.sql` can be e.g., `00001_removeTableX.down.sql` and should be interpreted as:
|
||||
- `00001`: The targeted `user_version` number.
|
||||
- `removeTableX`: What the script does.
|
||||
- `down`: This script `downgrade`s the database from `user_version = 00002` to the `user_version = 00001`.
|
||||
|
||||
There can be more than one migration file for the same `user_version`.
|
||||
The migration process will consider all such files while upgrading/downgrading the database.
|
||||
Note that currently we **DO NOT** support **down migration**.
|
||||
|
||||
# User Guide
|
||||
Migrations work out of the box.
|
||||
However, if you want to be extra sure, please take a backup of the SQLite database prior to upgrading your nim-waku version since we currently don't support downgrades of DB.
|
||||
Loading…
Reference in New Issue