waku-org
/
nwaku
mirror of https://github.com/waku-org/nwaku.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
nwaku/docs/tutorial/db-migration.md

63 lines
3.6 KiB
Markdown
Raw Normal View History

DB Migration tutorial (#646) * adds timestamp to waku message store impl * stores timestamp as int64 * adds untitest * stores timestamp as seq of bytes * minor * re-orders unittest * changes receiver timestamp to float64 * unit test for receiver timestamps * adds comments * reorder a few lines * updates changelog * more updates on changelog * WIP * WIP * adds migration * more debug messages * passes the path to the migration scripts from message store module * adds migration result type * replaces migrationScripts with migrationScriptsResult * adds path calculation to the message store * removes some tests binary file * removes redundant imports * comments out user_version assignment in sqlite init * more descriptive err messages * clean up test file * more info logs * minor code format * removes a todo * minor updates * remove a binary file * unit tests for migration utils * adds split script * integrates split query to handle scripts with multiple commands * updates migration script for v1 * updates the v1 migration script * update user version * updates script * fixes a few bugs on the splitScript * more debug logs * adds float64 parameter support to sqlite3 * change in timestamp type in the script * deletes float64 toBytes utils * enables storage of timestamp as a real number in the sqlite db * bump up script index * comment edits * removes migrate unit test * adds todo and docstring * updates changelog * removes an unused item in .gitignore * minor * updates changelog * organizes imports * cleans up imports * WIP * updates script fixes a few bugs on the splitScript more debug logs adds float64 parameter support to sqlite3 change in timestamp type in the script deletes float64 toBytes utils * edits migration util test * remove an empty test file * includes migration utils tests in * deletes unused codes * tides up imports * adds range based filter to the filterMigrationScripts * renames procs: removes Migration * tides up imports * edits docstring * edits docstring * edits docstring * removes unused imports * more clean up * groups std imports * updates changelog * adds docstring for setUserVersion * adds unittest for the migrate * Update waku/v2/node/storage/message/waku_message_store.nim Co-authored-by: RichΛrd <info@richardramos.me> * Update waku/v2/node/storage/sqlite.nim Co-authored-by: RichΛrd <info@richardramos.me> * Update waku/v2/node/storage/sqlite.nim Co-authored-by: RichΛrd <info@richardramos.me> * removes split scripts * fixes a naming issue * fixes a bug * fixes a typo * adds a log re updated user_version * fixes a proc naming mismatch * fixes naming mismatch * more descriptive var names * adds migration script of the first user version * moves migration to after persistMessages flag is checked * deletes unused comment * fixes a bug * brings back split script * adds unit tests for split scripts * runs scripts one command at a time * deletes a commented line * relocates the migrate proc to sqlite.nim * initial writeup * adds migration tutorial * adds User guides to db migration tutorial * minor * proofread * replaces a broken link * user-version to user_version * some clarification * some edits * minor * a quick wording fix * minor * final edits * updates the changelog * slight change Co-authored-by: RichΛrd <info@richardramos.me>
2021-06-25 18:35:46 +00:00
# Database Migration
This tutorial explains the database migration process in nim-waku.
# Contributors Guide
## Database Migration Flow
Nim-waku utilizes the built-in `user_version` variable that Sqlite provides for tracking the database versions.
The [user_version](https://github.com/status-im/nim-waku/blob/master/waku/v2/node/storage/migration/migration_types.nim) MUST be bumped up for every update on the database e.g, table schema/title change.
Each update should be accompanied by a migration script to move the content of the old version of the database to the new version.
The script MUST be added to the respective folder as explained in [Migration Folder Structure](#migration-folder-structure) with the proper naming as given in [ Migration Script Naming](#migration-file-naming).
The migration is invoked whenever the database `user_version` is behind the target [user_version](https://github.com/status-im/nim-waku/blob/master/waku/v2/node/storage/migration/migration_types.nim) indicated in the nim-waku application.
The respective migration scripts located in the [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) will be executed to upgrade the database from its old version to the target version.
## Migration Folder Structure
The [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) is structured as below.
```
|-- migration
| |--migration_scripts
| | |--message
| | | |--00001_basicMessageTable.up.sql
| | | |--00002_addSenderTimeStamp.up
| | | |-- ...
| | |--peer
| | | |--00001_basicPeerTable.up.sql
| | | |-- ...
```
The migration scripts are managed in two separate folders `message` and `peer`.
The `message` folder contains the migration scripts related to the message store tables.
Similarly, the `peer` folder contains the scripts relevant to the peer store tables.
## Migration File Naming
The files in [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) MUST follow the following naming style in order to be properly included in the migration process.
Files with invalid naming will be eliminated from the migration process.
`<version number>_<migration script description>.<up|down>.sql`
- `version number`: This number should match the target value of `user_version`.
- `migration script description`: A short description of what the migration script does.
- `up|down`: One of the keywords of `up` or `down` should be selected.
`up` stands for upgrade and `down` means downgrade.
### Example
A migration file with the name `00002_addTableX.up.sql` should be interpreted as:
- `00002`: The targeted `user_version` number.
- `addTableX`: What the script does.
- `up`: This script `upgrade`s the database from `user_version = 00001` to the `user_version = 00002`.
A downgrade migration file corresponding to the `00002_addTableX.up.sql` can be e.g., `00001_removeTableX.down.sql` and should be interpreted as:
- `00001`: The targeted `user_version` number.
- `removeTableX`: What the script does.
- `down`: This script `downgrade`s the database from `user_version = 00002` to the `user_version = 00001`.
There can be more than one migration file for the same `user_version`.
The migration process will consider all such files while upgrading/downgrading the database.
Note that currently we **DO NOT** support **down migration**.
# User Guide
Migrations work out of the box.
However, if you want to be extra sure, please take a backup of the SQLite database prior to upgrading your nim-waku version since we currently don't support downgrades of DB.