From 10f692a9a6134483261dc4e621604ca1aa849331 Mon Sep 17 00:00:00 2001 From: Sanaz Taheri Boshrooyeh <35961250+staheri14@users.noreply.github.com> Date: Fri, 25 Jun 2021 11:35:46 -0700 Subject: [PATCH] DB Migration tutorial (#646) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * adds timestamp to waku message store impl * stores timestamp as int64 * adds untitest * stores timestamp as seq of bytes * minor * re-orders unittest * changes receiver timestamp to float64 * unit test for receiver timestamps * adds comments * reorder a few lines * updates changelog * more updates on changelog * WIP * WIP * adds migration * more debug messages * passes the path to the migration scripts from message store module * adds migration result type * replaces migrationScripts with migrationScriptsResult * adds path calculation to the message store * removes some tests binary file * removes redundant imports * comments out user_version assignment in sqlite init * more descriptive err messages * clean up test file * more info logs * minor code format * removes a todo * minor updates * remove a binary file * unit tests for migration utils * adds split script * integrates split query to handle scripts with multiple commands * updates migration script for v1 * updates the v1 migration script * update user version * updates script * fixes a few bugs on the splitScript * more debug logs * adds float64 parameter support to sqlite3 * change in timestamp type in the script * deletes float64 toBytes utils * enables storage of timestamp as a real number in the sqlite db * bump up script index * comment edits * removes migrate unit test * adds todo and docstring * updates changelog * removes an unused item in .gitignore * minor * updates changelog * organizes imports * cleans up imports * WIP * updates script fixes a few bugs on the splitScript more debug logs adds float64 parameter support to sqlite3 change in timestamp type in the script deletes float64 toBytes utils * edits migration util test * remove an empty test file * includes migration utils tests in * deletes unused codes * tides up imports * adds range based filter to the filterMigrationScripts * renames procs: removes Migration * tides up imports * edits docstring * edits docstring * edits docstring * removes unused imports * more clean up * groups std imports * updates changelog * adds docstring for setUserVersion * adds unittest for the migrate * Update waku/v2/node/storage/message/waku_message_store.nim Co-authored-by: RichΛrd * Update waku/v2/node/storage/sqlite.nim Co-authored-by: RichΛrd * Update waku/v2/node/storage/sqlite.nim Co-authored-by: RichΛrd * removes split scripts * fixes a naming issue * fixes a bug * fixes a typo * adds a log re updated user_version * fixes a proc naming mismatch * fixes naming mismatch * more descriptive var names * adds migration script of the first user version * moves migration to after persistMessages flag is checked * deletes unused comment * fixes a bug * brings back split script * adds unit tests for split scripts * runs scripts one command at a time * deletes a commented line * relocates the migrate proc to sqlite.nim * initial writeup * adds migration tutorial * adds User guides to db migration tutorial * minor * proofread * replaces a broken link * user-version to user_version * some clarification * some edits * minor * a quick wording fix * minor * final edits * updates the changelog * slight change Co-authored-by: RichΛrd --- CHANGELOG.md | 1 + docs/tutorial/db-migration.md | 62 +++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) create mode 100644 docs/tutorial/db-migration.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 147062314..8709dfc5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ This release contains the following: #### General refactoring #### Docs +- Adds the database migration tutorial. #### Schema - Updates the `Message` table of the persistent message store: - Adds `senderTimestamp` column. diff --git a/docs/tutorial/db-migration.md b/docs/tutorial/db-migration.md new file mode 100644 index 000000000..819805d10 --- /dev/null +++ b/docs/tutorial/db-migration.md @@ -0,0 +1,62 @@ +# Database Migration +This tutorial explains the database migration process in nim-waku. + +# Contributors Guide +## Database Migration Flow +Nim-waku utilizes the built-in `user_version` variable that Sqlite provides for tracking the database versions. +The [user_version](https://github.com/status-im/nim-waku/blob/master/waku/v2/node/storage/migration/migration_types.nim) MUST be bumped up for every update on the database e.g, table schema/title change. +Each update should be accompanied by a migration script to move the content of the old version of the database to the new version. +The script MUST be added to the respective folder as explained in [Migration Folder Structure](#migration-folder-structure) with the proper naming as given in [ Migration Script Naming](#migration-file-naming). + +The migration is invoked whenever the database `user_version` is behind the target [user_version](https://github.com/status-im/nim-waku/blob/master/waku/v2/node/storage/migration/migration_types.nim) indicated in the nim-waku application. +The respective migration scripts located in the [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) will be executed to upgrade the database from its old version to the target version. + +## Migration Folder Structure +The [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) is structured as below. + +``` +|-- migration +| |--migration_scripts +| | |--message +| | | |--00001_basicMessageTable.up.sql +| | | |--00002_addSenderTimeStamp.up +| | | |-- ... +| | |--peer +| | | |--00001_basicPeerTable.up.sql +| | | |-- ... +``` + +The migration scripts are managed in two separate folders `message` and `peer`. +The `message` folder contains the migration scripts related to the message store tables. +Similarly, the `peer` folder contains the scripts relevant to the peer store tables. + + +## Migration File Naming +The files in [migrations folder](https://github.com/status-im/nim-waku/tree/master/waku/v2/node/storage/migration) MUST follow the following naming style in order to be properly included in the migration process. +Files with invalid naming will be eliminated from the migration process. + +`_..sql` + +- `version number`: This number should match the target value of `user_version`. +- `migration script description`: A short description of what the migration script does. +- `up|down`: One of the keywords of `up` or `down` should be selected. + `up` stands for upgrade and `down` means downgrade. + +### Example +A migration file with the name `00002_addTableX.up.sql` should be interpreted as: +- `00002`: The targeted `user_version` number. +- `addTableX`: What the script does. +- `up`: This script `upgrade`s the database from `user_version = 00001` to the `user_version = 00002`. + +A downgrade migration file corresponding to the `00002_addTableX.up.sql` can be e.g., `00001_removeTableX.down.sql` and should be interpreted as: +- `00001`: The targeted `user_version` number. +- `removeTableX`: What the script does. +- `down`: This script `downgrade`s the database from `user_version = 00002` to the `user_version = 00001`. + +There can be more than one migration file for the same `user_version`. +The migration process will consider all such files while upgrading/downgrading the database. +Note that currently we **DO NOT** support **down migration**. + +# User Guide +Migrations work out of the box. +However, if you want to be extra sure, please take a backup of the SQLite database prior to upgrading your nim-waku version since we currently don't support downgrades of DB.