mirror of https://github.com/logos-messaging/logos-messaging-nim.git synced 2026-01-03 06:23:10 +00:00

Sanaz Taheri Boshrooyeh 10f692a9a6 DB Migration tutorial (#646 )

* adds timestamp to waku message store impl

* stores timestamp as int64

* adds untitest

* stores timestamp as seq of bytes

* minor

* re-orders unittest

* changes receiver timestamp to float64

* unit test for receiver timestamps

* adds comments

* reorder a few lines

* updates changelog

* more updates on changelog

* WIP

* WIP

* adds migration

* more debug messages

* passes the path to the migration scripts from message store module

* adds migration result type

* replaces migrationScripts with migrationScriptsResult

* adds path calculation to the message store

* removes some tests binary file

* removes redundant imports

* comments out user_version assignment in sqlite  init

* more descriptive err messages

* clean up test file

* more info logs

* minor code format

* removes a todo

* minor updates

* remove a binary file

* unit tests for migration utils

* adds split script

* integrates split query to handle scripts with multiple commands

* updates migration script for v1

* updates the v1 migration script

* update user version

* updates script

* fixes a few bugs on the splitScript

* more debug logs

* adds float64 parameter support to sqlite3

* change in timestamp type in the script

* deletes float64 toBytes utils

* enables storage of timestamp as a real number in the sqlite db

* bump up script index

* comment edits

* removes migrate unit test

* adds todo and docstring

* updates changelog

* removes an unused item in .gitignore

* minor

* updates changelog

* organizes imports

* cleans up imports

* WIP

* updates script


fixes a few bugs on the splitScript


more debug logs


adds float64 parameter support to sqlite3


change in timestamp type in the script


deletes float64 toBytes utils

* edits migration util test

* remove an empty test file

* includes migration utils tests in

* deletes unused codes

* tides up imports

* adds range based filter to the filterMigrationScripts

* renames procs: removes Migration

* tides up imports

* edits docstring

* edits docstring

* edits docstring

* removes unused imports

* more clean up

* groups std imports

* updates changelog

* adds docstring for setUserVersion

* adds unittest for the migrate

* Update waku/v2/node/storage/message/waku_message_store.nim

Co-authored-by: RichΛrd <info@richardramos.me>

* Update waku/v2/node/storage/sqlite.nim

Co-authored-by: RichΛrd <info@richardramos.me>

* Update waku/v2/node/storage/sqlite.nim

Co-authored-by: RichΛrd <info@richardramos.me>

* removes split scripts

* fixes a naming issue

* fixes a bug

* fixes a typo

* adds a log re updated user_version

* fixes a proc naming mismatch

* fixes naming mismatch

* more descriptive var names

* adds migration script of the first user version

* moves migration to after persistMessages flag is checked

* deletes unused comment

* fixes a bug

* brings back split script

* adds unit tests for split scripts

* runs scripts one command at a time

* deletes a commented line

* relocates the migrate proc to sqlite.nim

* initial writeup

* adds migration tutorial

* adds User guides to db migration tutorial

* minor

* proofread

* replaces a broken link

* user-version to user_version

* some clarification

* some edits

* minor

* a quick wording fix

* minor

* final edits

* updates the changelog

* slight change

Co-authored-by: RichΛrd <info@richardramos.me>

2021-06-25 11:35:46 -07:00

8.7 KiB

Raw Blame History

Changelog

This release contains the following:

Features

Changes

Enables db migration for the message store.
The resume Nim API eliminates duplicates messages before storing them.

General refactoring

Docs

Adds the database migration tutorial.

Schema

Updates the Message table of the persistent message store:
- Adds senderTimestamp column.
- Renames the timestamp column to receiverTimestamp and changes its type to REAL.

API

Fixes

2021-06-03 v0.4

This release contains the following:

Features

Initial toy-chat implementation

Changes

The toy-chat application can now perform lightpush and request content-filtered messages from remote peers.
The toy-chat application now uses default content topic /toy-chat/2/huilong/proto
Improve toy-chat briding to matterbridge
Improve swap logging and enable soft mode by default
Content topics are no longer in a redundant nested structure
Improve error handling

API

JSON-RPC Store API: Added an optional time-based query to filter historical messages.
Nim API: Added resume method.

Fixes

Connections between nodes no longer become unstable due to keep-alive errors if mesh grows large
Re-enable lightpush tests and fix Windows CI failure

The Waku v2 suite of protocols are still in a raw/draft state. This release supports the following libp2p protocols:

Protocol	Spec status	Protocol id
`17/WAKU-RLN`	`raw`	`/vac/waku/waku-rln-relay/2.0.0-alpha1`
`11/WAKU2-RELAY`	`draft`	`/vac/waku/relay/2.0.0-beta2`
`12/WAKU2-FILTER`	`draft`	`/vac/waku/filter/2.0.0-beta1`
`13/WAKU2-STORE`	`draft`	`/vac/waku/store/2.0.0-beta3`
`18/WAKU2-SWAP`	`draft`	`/vac/waku/swap/2.0.0-beta1`
`19/WAKU2-LIGHTPUSH`	`draft`	`/vac/waku/lightpush/2.0.0-beta1`

The Waku v1 implementation is stable but not under active development.

2021-05-11 v0.3

This release contains the following:

Features

Start of RLN relay implementation
Start of swap implementation
Start of fault-tolerant store implementation
Initial bridge implementation between Waku v1 and v2 protocols
Initial lightpush implementation
A peer manager for relay, filter, store and swap peers
Persistent storage for peers: A node with this feature enabled will now attempt to reconnect to relay peers after a restart. It will respect the gossipsub PRUNE backoff period before attempting to do so.
--persist-peers CLI option to persist peers in local storage
--persist-messages CLI option to store historical messages locally
--keep-alive CLI option to maintain a stable connection to relay peers on idle topics
A CLI chat application (chat2) over Waku v2 with bridging to matterbridge

Changes

Enable swap protocol by default and improve logging

General refactoring

Split out waku_types types into the right place; create utils folder.
Change type of contentTopic in ContentFilter to string.
Replace sequence of contentTopics in ContentFilter with a single contentTopic.
Add timestamp field to WakuMessage.
Ensure CLI config parameters use a consistent naming scheme. Summary of changes here.

Docs

Several clarifications and additions aimed at contributors, including

information on how to query Status test fleet for node addresses,
how to view logs, and
how to update submodules.

Schema

Add Message table to the persistent message store. This table replaces the old messages table. It has two additional columns, namely
- pubsubTopic, and
- version.
Add Peer table for persistent peer storage.

API

JSON-RPC Admin API: Added a post method to connect to peers on an ad-hoc basis.
Nim API: PubSub topic subscribe and unsubscribe no longer returns a future (removed async designation).
HistoryQuery: Added pubsubTopic field. Message history can now be filtered and queried based on the pubsubTopic.
HistoryQuery: Added support for querying a time window by specifying start and end times.

Fixes

Running nodes can now be shut down gracefully
Content filtering now works on any PubSub topic and not just the waku default.
Nodes can now mount protocols without supporting relay as a capability

The Waku v2 suite of protocols are still in a raw/draft state. This release supports the following libp2p protocols:

Protocol	Spec status	Protocol id
`17/WAKU-RLN`	`raw`	`/vac/waku/waku-rln-relay/2.0.0-alpha1`
`18/WAKU2-SWAP`	`raw`	`/vac/waku/swap/2.0.0-alpha1`
`19/WAKU2-LIGHTPUSH`	`raw`	`/vac/waku/lightpush/2.0.0-alpha1`
`11/WAKU2-RELAY`	`draft`	`/vac/waku/relay/2.0.0-beta2`
`12/WAKU2-FILTER`	`draft`	`/vac/waku/filter/2.0.0-beta1`
`13/WAKU2-STORE`	`draft`	`/vac/waku/store/2.0.0-beta3`

The Waku v1 implementation is stable but not under active development.

2021-01-05 v0.2

This release contains the following changes:

Calls to publish a message on wakunode2 now await instead of discard dispatched WakuRelay procedures.
StrictNoSign enabled.
Add JSON-RPC API for external access to wakunode2 functionality:
- Admin API retrieves information about peers registered on the wakunode2.
- Debug API exposes debug information about a wakunode2.
- Filter API saves bandwidth by allowing light nodes to filter for specific content.
- Private API enables symmetric or asymmetric cryptography to encrypt/decrypt message payloads.
- Relay API allows basic pub/sub functionality.
- Store API retrieves historical messages.
Add tutorial on how to use JSON-RPC API.
Refactor: Move waku_filter protocol into its own module.

The Waku v2 implementation, and most protocols it consist of, are still in a draft/beta state. The Waku v1 implementation is stable but not under active development.

2020-11-30 v0.1

Initial beta release.

This release contains:

A Nim implementation of the Waku v1 protocol.
A Nim implementation of the Waku v2 protocol.
CLI applications wakunode and wakunode2 that allows you to run a Waku v1 or v2 node.
Examples of Waku v1 and v2 usage.
Various tests of above.

Currenty the Waku v2 implementation, and most protocols it consist of, are in a draft/beta state. The Waku v1 implementation is stable but not under active development.

Feedback welcome!

8.7 KiB Raw Blame History

Changelog

Next

Features

Changes

General refactoring

Docs

Schema

API

Fixes

2021-06-03 v0.4

Features

Changes

API

Fixes

2021-05-11 v0.3

Features

Changes

General refactoring

Docs

Schema

API

Fixes

2021-01-05 v0.2

2020-11-30 v0.1

8.7 KiB

Raw Blame History