* chore(rln): rln message limit to 100 (#2883)
* postgres_driver: add more error handling when creating partitions
Given that multiple nodes can be connected to the same database,
it can happen that other node did something that my node was willing
to do. In this commit, we overcome the possible "interleaved"
partition creation.
---------
Co-authored-by: Alvaro Revuelta <alvrevuelta@gmail.com>
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
The id column is needed because it contains the message digest
which is used in store v2, and we need to keep support to
store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
We want the nwaku node to simultaneously support store-v2
requests and store-v3 requests.
Only the legacy archive is in charge of archiving messages, and the
archived information is suitable to fulfill both store-v2 and
store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
With the original approach it happened cases where one connection
acquired the lock and other connection tried to release the same lock,
causing an unrecoverable failure which made the node to end.
* archive.nim: reduce the database report interval from 1 to 30 min
This counts the number of rows with "select count(1) from messages"
which is quite intense and we shouldn't run it every minute
* aside cleanup
* chore: extract select to constant for multiple prefix
* chore: Add a space at the end of selectClause to facilitate better string merging.
Signed-off-by: DarshanBPatel <darshan@status.im>
---------
Signed-off-by: DarshanBPatel <darshan@status.im>
* postgres_driver: add ON CONFLICT DO NOTHING in the insert statement
* test_driver_postgres: adapt test to ON CONFLICT DO NOTHING
The insert does not fail when duplicate, it returns a positive response
when doing 'put' of a duplicated row. The test is adapted so that
we just check that the number of messages doesn't grow after
trying to add a duplicated row.
* postgres: first step to implement partition management
* postgres_driver: use of times.now().toTime().toUnix() instead of Moment.now()
* postgres migrations: set new version to 2
* test_driver_postgres: use of assert instead of require and avoid using times.now()
* postgres_driver: better implementation of the reset method with partitions
* Remove createMessageTable, init, and deleteMessageTable procs
* postgres: ensure we use the version 15.4 in tests
* postgres_driver.nim: enhance debug logs partition addition
* ci.yml: ensure logs are printed without colors
* postgres_driver: starting the loop factory in an asynchronous task
* postgres_driver: log partition name and size when removing a partition
* Add postgres_driver/migrations.nim
* Postgres and archive logic adaptation to the migration implementation
* libwaku: adapt node_lifecycle_request.nim to migration refactoring
* test_app.nim: add more detail for test that only fails in CI
* postgres migrations: store the migration scripts inside the resulting wakunode binary instead of external .sql files.
* new proc to foster different size retention policy implementations
The new proc, decreaseDatabaseSize, will have different implementations
per each driver. For example, in future commits we will implement a size
retention policy thanks to partitions management, in Postgres.
* RetentionPolicy: use of new instead of init for ref object types
* waku_archive: fix signatures in decreaseDatabaseSize methods
* retention_policy_size: minor cleanup of comments and imports
* archive: move error to trace level when insert row fails
That is helpful to prevent the node to spam the logs when it shares
connection to the same Postgres database with other nodes, in
which case the following log appears too much:
topics="waku archive" tid=1 file=archive.nim:113 err="error in
runStmt: error in dbConnQueryPrepared calling waitQueryToFinish: error
in query: ERROR: duplicate key value violates unique constraint
"messageindex" DETAIL: Key
(messagehash)=(88f4ee115eef6f233a7dceaf975f03946e18666adda877e38d61be98add934e8)
already exists. "
Before this commit, the following execution of a prepared statement
returned nothing even though the database had 2 rows to be returned:
nwaku-db-1 | 2023-12-14 12:55:17.575 UTC [73] LOG: execute SelectWithoutCursorAsc: SELECT storedAt, contentTopic, payload, pubsubTopic, version, timestamp, id FROM messages
nwaku-db-1 | WHERE contentTopic IN ($1) AND
nwaku-db-1 | pubsubTopic = $2 AND
nwaku-db-1 | storedAt >= $3 AND
nwaku-db-1 | storedAt <= $4
nwaku-db-1 | ORDER BY storedAt ASC LIMIT $5;
nwaku-db-1 | 2023-12-14 12:55:17.575 UTC [73] DETAIL: parameters: $1 =
'my/ctopic/1,my/ctopic/2', $2 = '/waku/2/default-waku/proto', $3 = '1702552968570786800', $4 = '1702552968585347557', $5 = '101'
The reason why it is not returning anything is that the 'IN' statement doesn't work when using prepared statements with multiple items. It only works when the 'IN' content, i.e. $1, contains one single item.
* on_chain/group_manager: use .async: (raises:[Exception]).
* bump nim-dnsdisc
* update nim-chronos to the latest state
* chat2.nim: catch any possible exception when stopping
* chat2bridge.nim: make it to compile after vendor bump
* ValidIpAddress (deprecated) -> IpAddress
* vendor/nim-libp2p additional bump
* libwaku: adapt to vendor bump
* testlib/wakunode.nim: adapt to vendor bump (ValidIpAddress -> IpAddress)
* waku_node: avoid throwing any exception from stop*(node: WakuNode)
* test_confutils_envvar.nim: ValidIpAddress -> IpAddress
* test_jsonrpc_store: capture exception
* test_rln*: handling exceptions
* adaptation to make test_rln_* to work properly
* signature enhancement of group_manager methods
