* feat(sqlite): parameterized prep statement exec
- from nim-eth
* feat(store): sql-only-store
- add getPage(...) to sql-message-store
- acts as separate store impl that can be activated with
`--sqlite-store=true`
* test(store): new test-suite for sql message store
* test(store): waku_store test SQL-store compatible
* perf(store): avoid builing an additional B-tree in sql query
- use `receiverTime` as the last ORDER BY column to fully utilize the
primary key sorting
feat(store): retention time
- retention time in seconds can be specified via `--sqlite-retention-time=`
* fix(store): limit capacity of persistent message store
fixes performance issue of loading a large message table by deleting
old messages in case of an overflow
* test(store): message capacity limit
* Refactor timestamps type from float64 to int64 (milliseconds resolution)
* Revert epochs to float64
* Update 00002_addSenderTimeStamp.up.sql
* Update quicksim2.nim
* Add files via upload
* Delete 00003_convertTimestampsToInts.up.sql
* Add files via upload
* Rename 00003_convertTimestampsToInts.up.sql to 00003_addTimestampsToInts.up.sql
* Delete 00003_addTimestampsToInts.up.sql
* Rln-relay integration into chat2 (#835)
* adds ProofMetadata
* adds EPOCH_INTERVAL
* adds messageLog field
* adds updateLog, toEpoch, fromEpoch, getEpoch, compareTo
* adds unit test for toEpoch and fromEpoch
* adds unit test for Epoch comparison
* adds result codes for updateLog
* adds unit test for update log
* renames epoch related consts
* modifies updateLog with new return type and new logic of spam detection
* adds unit text for the modified updateLog
* changes max epoch gap type size
* splits updateLog into two procs isSpam and updateLog
* updates unittests
* fixes a bug, returns false when the message is not spam
* renames messageLog to nullifierLog
* renames isSpam to hasDuplicate
* updates the rln validator, adds comments
* adds appendRLNProof proc plus some code beatification
* unit test for validate message
* adds unhappy test to validateMessage unit test
* renames EPOCH_UNIT_SECONDS
* renames MAX_CLOCK_GAP_SECONDS
* WIP: integration test
* fixes compile errors
* sets a real epoch value
* updates on old unittests
* adds comments to the rln relay tests
* adds more comments
* makes rln import conditional
* adds todos
* adds more todos
* adds rln-relay mount process into chat2
* further todos
* logs contentTopic
* introduces rln relay configs
* changes default pubsub topic
* adds contentTopic config
* imports rln relay dependencies
* consolidates imports
* removes module identifier from ContentTopic
* adds contentTopic field
* adds contentTopic argument to mountRlnRelay calls
* appends rln proof to chat2 messages
* changes the default chat2 contentTopic
* adds missing content topic fields
* fixes a bug
* adds a new logic about empty content topics
* appends proof only when rln flag is active
* removes unnecessary todos
* fixes an indentation issue
* adds log messages
* verifies the proof against the concatenation of msg payload and content topic
* a bug fix
* removes duplicate epoch time calculation
* updates log level to trace
* updates default rln-relay content topic
* adds support for empty content topics
* updates changelog
* changelog updates
* removes a commented code block
* updates addRLNRelayValidator string doc
* Squashed commit of the following:
commit bc36c99ab202d07baa0a5f0100bd10d1d76fdfa1
Merge: dc2b2946 5a77d6e2
Author: G <28568419+s1fr0@users.noreply.github.com>
Date: Sat Feb 5 01:10:06 2022 +0100
Merge branch 'master' into int64-timestamps-ns
commit dc2b294667bb5770cc32b93cc560638cf5ce7087
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Sat Feb 5 00:24:45 2022 +0100
Fix
commit f97b95a036a197938df38a5adaea46fca778016d
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Sat Feb 5 00:13:18 2022 +0100
Missing import
commit 060c4f8d64e1b6e7c0593540fa8fa7f4cadf6df7
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Sat Feb 5 00:10:36 2022 +0100
Fixed typo
commit 08ca99b6f692d3df6d4c7c2312c7cada05fc0041
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 23:59:20 2022 +0100
Time util file
commit 2b5c360746990936dec256e90d08dae3c3e35a94
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 23:33:20 2022 +0100
Moved time utility functions to utils/time
commit fdaf121f089aa011855303cc8dd1ce52aec506ad
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 23:10:25 2022 +0100
Fix comment
commit c7e06ab4e7618d9a3fe8aa744dd48bf3f7d8754c
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 23:04:13 2022 +0100
Restore previous migration script
commit 80282db1d79df676255d4b8e6e09d9f8a2b00fd3
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 22:54:15 2022 +0100
Typo
commit b9d67f89b0eea11a8362dbb10b5f9d6894343352
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 22:49:29 2022 +0100
Added utilities to get int64 nanosecond, microsecond, millisecond time resolution from float
commit 0130d496e694a01cfc9eeb90b7cbc77764490bf9
Author: s1fr0 <28568419+s1fr0@users.noreply.github.com>
Date: Fri Feb 4 22:36:35 2022 +0100
Switched to nanoseconds support.
* Update CHANGELOG.md
* Create 00003_convertTimestampsToInt64.up.sql
Migration script
* Moved migration script to right location
* Update waku_rln_relay_utils.nim
* Update waku_rln_relay_utils.nim
* Addressed reviewers' comments
* Update default fleet metrics dashboard (#844)
* Fix
* No need for float
* Aligning master to changes in PR
* Further fixes
Co-authored-by: Sanaz Taheri Boshrooyeh <35961250+staheri14@users.noreply.github.com>
Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
* enables pagination based on sender timestamp
* uncomments a test
* bumps up version number
* updates migration script
* clean up
* unpdates changelog
* undo removal of receiver timestamp
* updates message_storage
* uses epochTime()
* minor
* removes a comment
* removes receiver timestamp deletion migration script
* fixes formatting issues
* fixes a bad field name
* fixes field issue
* adjusts spacing
* moves migration paths to the migration_types module
* moves USER_VERSION to the migration_types module
* updates migration execution condition to cover the peer store
* adds a message to the store if it is inserted to the db
* adds debug messages and checks migration path recursively
* deletes unused debug messages
* deletes commented codes
* clean up
* fixes a bug
* renames the first migration file of the message store
* adds timestamp to waku message store impl
* stores timestamp as int64
* adds untitest
* stores timestamp as seq of bytes
* minor
* re-orders unittest
* changes receiver timestamp to float64
* unit test for receiver timestamps
* adds comments
* reorder a few lines
* updates changelog
* more updates on changelog
* WIP: deduplicate
* add debug messages
* more debugging logs
* add timestamp to the message store
* converts resume result to uint64, removes some logs
* docstring for isDuplicate
* increments `added` before insertion into the db
* unit test for deduplication
* removes float to/from Bytes procs
* deletes extra spaces
* returns failed instead of the error
* minor
* minor
* adds changelog
* adds timestamp to waku message store impl
* stores timestamp as int64
* adds untitest
* stores timestamp as seq of bytes
* minor
* re-orders unittest
* changes receiver timestamp to float64
* unit test for receiver timestamps
* adds comments
* reorder a few lines
* updates changelog
* more updates on changelog
* WIP
* WIP
* adds migration
* more debug messages
* passes the path to the migration scripts from message store module
* adds migration result type
* replaces migrationScripts with migrationScriptsResult
* adds path calculation to the message store
* removes some tests binary file
* removes redundant imports
* comments out user_version assignment in sqlite init
* more descriptive err messages
* clean up test file
* more info logs
* minor code format
* removes a todo
* minor updates
* remove a binary file
* unit tests for migration utils
* adds split script
* integrates split query to handle scripts with multiple commands
* updates migration script for v1
* updates the v1 migration script
* update user version
* updates script
* fixes a few bugs on the splitScript
* more debug logs
* adds float64 parameter support to sqlite3
* change in timestamp type in the script
* deletes float64 toBytes utils
* enables storage of timestamp as a real number in the sqlite db
* bump up script index
* comment edits
* removes migrate unit test
* adds todo and docstring
* updates changelog
* removes an unused item in .gitignore
* minor
* updates changelog
* organizes imports
* cleans up imports
* WIP
* updates script
fixes a few bugs on the splitScript
more debug logs
adds float64 parameter support to sqlite3
change in timestamp type in the script
deletes float64 toBytes utils
* edits migration util test
* remove an empty test file
* includes migration utils tests in
* deletes unused codes
* tides up imports
* adds range based filter to the filterMigrationScripts
* renames procs: removes Migration
* tides up imports
* edits docstring
* edits docstring
* edits docstring
* removes unused imports
* more clean up
* groups std imports
* updates changelog
* adds docstring for setUserVersion
* adds unittest for the migrate
* Update waku/v2/node/storage/message/waku_message_store.nim
Co-authored-by: RichΛrd <info@richardramos.me>
* Update waku/v2/node/storage/sqlite.nim
Co-authored-by: RichΛrd <info@richardramos.me>
* Update waku/v2/node/storage/sqlite.nim
Co-authored-by: RichΛrd <info@richardramos.me>
* removes split scripts
* fixes a naming issue
* fixes a bug
* fixes a typo
* adds a log re updated user_version
* fixes a proc naming mismatch
* fixes naming mismatch
* more descriptive var names
* adds migration script of the first user version
* moves migration to after persistMessages flag is checked
* deletes unused comment
* fixes a bug
* brings back split script
* adds unit tests for split scripts
* runs scripts one command at a time
* deletes a commented line
* relocates the migrate proc to sqlite.nim
* adds unit test for filter scripts
* adds filterScripts unittest testing varying zero-prefixed user versions
* minor
Co-authored-by: RichΛrd <info@richardramos.me>
* WIP
* stores version in db, edits error messages
* fixes a bug
* reverts edit messages
* converts uint32 storage to int64
* unit tests for various version numbers
* minor comments edit
* removes debugging messages
* updates changelog
* replaces topics with seq of ContentFilters
* update topics to contentFilter
* updates the contentFilter structure
one content topic per content filter instead of a sequence of topics
* updates store json rpc api
* renames ContentFilter to HistoryContentFilter
* unit test for a query with several content filters
* makes shortcut for store api
* updates chat2
* clean up
* renames topic to contentTopic
* adds pubsub topic to the history query
updates message store interface to return the pubsub topic
updates waku message store implementation
updates database schema to hold pubsub topi per waku message
* clarifies the use of content topic in store api
* clarifies the use of contentTopic in the init method of HistoryContentFilter
* simplifies the test and add comments
* lowers the field number of pubsub topic in historyQuery protobuf
* captures an empty contentFilter case
* test pubsub topic filter for the entire history and no message match
* demoves duplicates
* adds TODO
* fix a broken comment line
* updates waku store codec
* swaps the order of pubsub topic and content topic in protobuf
* Update waku/v2/protocol/waku_store/waku_store_types.nim
Co-authored-by: Oskar Thorén <ot@oskarthoren.com>
* updates the pubsub topic to the default value
* bumps protocol id
* moves the comment close to IndexedWakuMessage
* adds checks to the store put method
* makes table title a constant variable and retitles the table to Message
* updates the changelog
* minor update
* minor
* beta2 to beta3
* minor
Co-authored-by: Oskar Thorén <ot@oskarthoren.com>
