Commit Graph

22 Commits

Author SHA1 Message Date
Ivan FB a0ee3d78fe
postgres_driver.nim: restrict getMessages prepared stmt to query with 1 content topic (#2296)
Before this commit, the following execution of a prepared statement
returned nothing even though the database had 2 rows to be returned:

nwaku-db-1  | 2023-12-14 12:55:17.575 UTC [73] LOG:  execute SelectWithoutCursorAsc: SELECT storedAt, contentTopic, payload, pubsubTopic, version, timestamp, id FROM messages
nwaku-db-1  | 	    WHERE contentTopic IN ($1) AND
nwaku-db-1  | 	          pubsubTopic = $2 AND
nwaku-db-1  | 	          storedAt >= $3 AND
nwaku-db-1  | 	          storedAt <= $4
nwaku-db-1  | 	    ORDER BY storedAt ASC LIMIT $5;
nwaku-db-1  | 2023-12-14 12:55:17.575 UTC [73] DETAIL:  parameters: $1 =
'my/ctopic/1,my/ctopic/2', $2 = '/waku/2/default-waku/proto', $3 = '1702552968570786800', $4 = '1702552968585347557', $5 = '101'

The reason why it is not returning anything is that the 'IN' statement doesn't work when using prepared statements with multiple items. It only works when the 'IN' content, i.e. $1, contains one single item.
2023-12-14 15:37:12 +01:00
Ivan FB b31c182325
chore: postres driver allow setting the max number of connection from a parameter (#2246)
* postres driver: allow setting the max number of connections from a parameter
2023-11-24 16:21:22 +01:00
Abhimanyu a1ed517f9c
fix: extended Postgres code to support retention policy + refactoring (#2244)
* updated Postgres retention policy code + refactoring

* Update waku/waku_archive/driver/postgres_driver/postgres_driver.nim

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>

* updated code review changes

* data unit fixed, processing everything in bytes now

---------

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>
2023-11-24 15:43:47 +01:00
Abhimanyu aeb77a3e75
feat: Add new DB column `messageHash` (#2202)
* feat: added DB column messageHash

* feat: minor change

* feat: minor merge conflict fix

* Update test_resume.nim

* Update test_resume.nim

* randomblob() func used to populate attribute

* PRIMARY key updated - SQLite and Postgres
2023-11-22 17:32:56 +01:00
Abhimanyu d7ef3ca192
Revert "feat: amending computeDigest func. + related test cases (#2132)" (#2180)
This reverts commit 1669f710ce.
2023-11-08 01:41:23 +01:00
Ivan FB 6da1aeec53
chore: Optimize postgres - prepared statements in select (#2182)
* db_postgres: use prepared statements on most freq select queries
* db_postgres/dbconn.nim adding better feedback in case of query error
* dbconn: use of isOkOr
* pgasyncpool: refactor to reduce code (valueOr, catch:)
2023-11-07 13:38:37 +01:00
Ivan FB 2b4ca4d0ff
chore: Optimize postgres - use of rowCallback approach (#2171)
* db_postgres, postgres_driver: better performance by using callback.
  There were a bunch of milliseconds being lost due to multiple-row
  processing. This commit aims to have the minimum possible row
  process time.
* pgasyncpool: clarifying logic around pool conn management.
* db_postgres: removing duplicate code and more searchable proc names.
2023-10-31 14:46:46 +01:00
Ivan FB 282c2e8107
chore: Minor Postgres optimizations (#2166)
* postgres_healthcheck: validate once per minute instead of 30 sec
* postgres_driver.nim: change MaxNumCons from 5 to 50
* postgres_driver.nim: split connPool into writeConPool and readConPool
  This aims to avoid clashes in insert and select queries
  because the inserts and selects can happen concurrently
  in relay and store events, respectively.
2023-10-30 15:16:49 +01:00
Abhimanyu 13aeebe46f
Revert "feat: messageHash attribute added in SQLite + testcase (#2142)" (#2154)
This reverts commit 9cd8c73d27.
2023-10-24 16:05:39 +02:00
Abhimanyu 9cd8c73d27
feat: messageHash attribute added in SQLite + testcase (#2142)
* feat: messageHash attaribute added in SQLite + testcase

* Update tests/waku_archive/test_driver_sqlite_query.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2023-10-24 12:19:52 +02:00
Abhimanyu 1669f710ce
feat: amending computeDigest func. + related test cases (#2132)
* feat: amending computeDigest func. + related test cases

* minor fixes

* minor fixes v1: testcase saga continues

---------

Co-authored-by: Vaclav Pavlin <vaclav@status.im>
2023-10-19 11:59:17 +02:00
Abhimanyu 25d6e52e38
chore: added size based retention policy (#2098)
* chore: added size based retention policy

* chore: post review code - size based retention policy

* chore: review integrated, size retention updated, vacuuming in capacity retention

* chore: typo fixed

* chore: review integrated

* chore: external config updated to support newly added retention policy
2023-10-10 15:29:09 +05:30
Abhimanyu 06bc433a80
fix: sqlite limited delete query bug (#2111) 2023-10-09 16:36:40 +05:30
Alvaro Revuelta a0033dff52
Revert "postgres_driver.nim: rename table's name from "messages" to "message" (#2110)" (#2115)
This reverts commit 71cfbbca6f.
2023-10-06 10:36:00 +02:00
Ivan FB 71cfbbca6f
postgres_driver.nim: rename table's name from "messages" to "message" (#2110) 2023-10-05 18:23:19 +02:00
Ivan FB c74abe131d
postgres_driver: remove duplicated const MaxNumConns. (#2106) 2023-10-04 15:28:07 +02:00
Hanno Cornelius c019016545
Revert "chore: add size retention policy (#2093)" (#2097)
This reverts commit 8897ae1a2f.

(cherry picked from commit b213b2c385b0534481448cd6e30af18e183d0504)
2023-10-02 11:07:13 +02:00
Abhimanyu 8897ae1a2f
chore: add size retention policy (#2093)
* chore: add retention policy with GB or MB limitation #1885

* chore: add retention policy with GB or MB limitation

* chore: updated code post review- retention policy

* ci: extract discordNotify to separate file

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: push images to new wakuorg/nwaku repo

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: enforce default Docker image tags strictly

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: push GIT_REF if it looks like a version

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* fix: update wakuv2 fleet DNS discovery enrtree

https://github.com/status-im/infra-misc/issues/171

* chore: resolving DNS IP and publishing it when no extIp is provided (#2030)

* feat(coverage): Add simple coverage (#2067)

* Add test aggregator to all directories.
* Implement coverage script.

* fix(ci): fix name of discord notify method

Also use absolute path to load Groovy script.

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version (#2080)

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version

* add more metrics, refactor how most metrics are calculated

* rework metrics table fillup

* reset connErr to make sure we honour successful reconnection

* chore(cbindings): Adding cpp example that integrates the 'libwaku' (#2079)

* Adding cpp example that integrates the `libwaku`

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>

* fix(ci): update the dependency list in pre-release WF (#2088)

* chore: adding NetConfig test suite (#2091)

---------

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Anton Iakimov <yakimant@gmail.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>
Co-authored-by: Vaclav Pavlin <vaclav@status.im>
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2023-09-30 11:10:52 +05:30
Ivan Folgueira Bande aa3e1a66d5
fix(sqlite): Properly set user_version to 7 so that the migration procedure is not started (#2031) 2023-09-14 14:12:37 +02:00
Ivan Folgueira Bande e8602021b6
chore(postgres): not loading the libpq library by default & better user feedback (#2028)
* removing implicit dependency with libpq if postgres is not being used.
* We only run the postgres tests when explicitly willing to, i.e. make
POSTGRES=1 test. The reason is that the postgres tests expect a
database instance to be running locally.
2023-09-13 12:45:55 +02:00
Ivan Folgueira Bande 1fb13b0967
Adding healtcheck and reconnection mechanism to the postgres archive driver (#1997)
It starts an asynchronous infinite task that checks the connectivity
with the database. In case of error, the postgres_healthcheck task
tries to reconnect for a while, and if it determines that the connection
cannot be resumed, then it invokes a callback indicating that
situation. For the case of the `wakunode2` app, this callback
quits the application itself and adds a log trace indicating
the connectivity issue with the database.
2023-09-06 19:16:37 +02:00
Hanno Cornelius b9d5d28af5
chore: remove references to v2 (#1898)
* chore: remove references to v2

* fix: lingering rln-relay import path
2023-08-09 18:11:50 +01:00