mirror of https://github.com/waku-org/nwaku.git synced 2025-01-10 23:06:34 +00:00

Sanaz Taheri Boshrooyeh 05cb073a0f

* adds timestamp to waku message store impl

* stores timestamp as int64

* adds untitest

* stores timestamp as seq of bytes

* minor

* re-orders unittest

* changes receiver timestamp to float64

* unit test for receiver timestamps

* adds comments

* reorder a few lines

* updates changelog

* more updates on changelog

* WIP

* WIP

* adds migration

* more debug messages

* passes the path to the migration scripts from message store module

* adds migration result type

* replaces migrationScripts with migrationScriptsResult

* adds path calculation to the message store

* removes some tests binary file

* removes redundant imports

* comments out user_version assignment in sqlite  init

* more descriptive err messages

* clean up test file

* more info logs

* minor code format

* removes a todo

* minor updates

* remove a binary file

* unit tests for migration utils

* adds split script

* integrates split query to handle scripts with multiple commands

* updates migration script for v1

* updates the v1 migration script

* update user version

* updates script

* fixes a few bugs on the splitScript

* more debug logs

* adds float64 parameter support to sqlite3

* change in timestamp type in the script

* deletes float64 toBytes utils

* enables storage of timestamp as a real number in the sqlite db

* bump up script index

* comment edits

* removes migrate unit test

* adds todo and docstring

* updates changelog

* removes an unused item in .gitignore

* minor

* updates changelog

* organizes imports

* cleans up imports

* WIP

* updates script


fixes a few bugs on the splitScript


more debug logs


adds float64 parameter support to sqlite3


change in timestamp type in the script


deletes float64 toBytes utils

* edits migration util test

* remove an empty test file

* includes migration utils tests in

* deletes unused codes

* tides up imports

* adds range based filter to the filterMigrationScripts

* renames procs: removes Migration

* tides up imports

* edits docstring

* edits docstring

* edits docstring

* removes unused imports

* more clean up

* groups std imports

* updates changelog

* adds docstring for setUserVersion

* adds unittest for the migrate

* Update waku/v2/node/storage/message/waku_message_store.nim

Co-authored-by: RichΛrd <info@richardramos.me>

* Update waku/v2/node/storage/sqlite.nim

Co-authored-by: RichΛrd <info@richardramos.me>

* Update waku/v2/node/storage/sqlite.nim

Co-authored-by: RichΛrd <info@richardramos.me>

* removes split scripts

* fixes a naming issue

* fixes a bug

* fixes a typo

* adds a log re updated user_version

* fixes a proc naming mismatch

* fixes naming mismatch

* more descriptive var names

* adds migration script of the first user version

* moves migration to after persistMessages flag is checked

* deletes unused comment

* fixes a bug

* brings back split script

* adds unit tests for split scripts

* runs scripts one command at a time

* deletes a commented line

* relocates the migrate proc to sqlite.nim

* initial writeup

* adds migration tutorial

* adds User guides to db migration tutorial

* minor

* proofread

* replaces a broken link

* user-version to user_version

* some clarification

* some edits

* minor

* a quick wording fix

* minor

* final edits

* updates the changelog

* slight change

Co-authored-by: RichΛrd <info@richardramos.me>

2021-06-25 11:35:46 -07:00

3.6 KiB

Raw Blame History

Database Migration

This tutorial explains the database migration process in nim-waku.

Contributors Guide

Database Migration Flow

Nim-waku utilizes the built-in user_version variable that Sqlite provides for tracking the database versions. The user_version MUST be bumped up for every update on the database e.g, table schema/title change. Each update should be accompanied by a migration script to move the content of the old version of the database to the new version. The script MUST be added to the respective folder as explained in Migration Folder Structure with the proper naming as given in Migration Script Naming.

The migration is invoked whenever the database user_version is behind the target user_version indicated in the nim-waku application.
The respective migration scripts located in the migrations folder will be executed to upgrade the database from its old version to the target version.

Migration Folder Structure

The migrations folder is structured as below.

|-- migration
|  |--migration_scripts
|  |  |--message
|  |  |  |--00001_basicMessageTable.up.sql
|  |  |  |--00002_addSenderTimeStamp.up
|  |  |  |-- ...
|  |  |--peer
|  |  |  |--00001_basicPeerTable.up.sql
|  |  |  |-- ...

The migration scripts are managed in two separate folders message and peer. The message folder contains the migration scripts related to the message store tables. Similarly, the peer folder contains the scripts relevant to the peer store tables.

Migration File Naming

The files in migrations folder MUST follow the following naming style in order to be properly included in the migration process. Files with invalid naming will be eliminated from the migration process.

<version number>_<migration script description>.<up|down>.sql

version number: This number should match the target value of user_version.
migration script description: A short description of what the migration script does.
up|down: One of the keywords of up or down should be selected. up stands for upgrade and down means downgrade.

Example

A migration file with the name 00002_addTableX.up.sql should be interpreted as:

00002: The targeted user_version number.
addTableX: What the script does.
up: This script upgrades the database from user_version = 00001 to the user_version = 00002.

A downgrade migration file corresponding to the 00002_addTableX.up.sql can be e.g., 00001_removeTableX.down.sql and should be interpreted as:

00001: The targeted user_version number.
removeTableX: What the script does.
down: This script downgrades the database from user_version = 00002 to the user_version = 00001.

There can be more than one migration file for the same user_version. The migration process will consider all such files while upgrading/downgrading the database. Note that currently we DO NOT support down migration.

User Guide

Migrations work out of the box. However, if you want to be extra sure, please take a backup of the SQLite database prior to upgrading your nim-waku version since we currently don't support downgrades of DB.

3.6 KiB Raw Blame History