46 Commits

Author SHA1 Message Date
Ivan FB
7e5041d5e1
Move log level from debug to info (#3622)
* convert all debug logs to info log level
* waku_relay protocol mv notice spammy logs to debug
2025-10-15 10:49:36 +02:00
Miran
ed0474ade3 chore: fix unused and deprecated imports (#3368) 2025-04-11 18:20:23 +03:00
Simon-Pierre Vivier
7dbc1fe061 fix: apply latest nph formating (#3334) 2025-03-19 11:30:47 -04:00
Ivan FB
564b6466a8 chore: better implementation to properly convert database query metrics (#3314) 2025-03-04 19:14:31 +01:00
Ivan FB
f65bea0f8e Revert "chore: waku_archive add protection against queries longer than 24h" (#3278)
This reverts commit 401402368d9075f93692d180cb30156785eed5a8.
2025-02-06 17:44:12 +01:00
Ivan FB
401402368d chore: waku_archive add protection against queries longer than 24h (#3256)
* store test adaptations because the tests were using future times
* waku_store_sync.nim, test_waku_archive, test_rln_group_manager_onchain.nim nph (unrelated change)
2025-01-27 10:44:59 +01:00
gabrielmer
4d9e11f16b chore: adding extra migration to sqlite and improving error message (#3240) 2025-01-16 16:10:28 +01:00
Ivan FB
e128385e69 chore: Optimize store (#3061)
* use messages_lookup to retrieve timestamps
* deep refactoring in db_postgres for better use of async approach
2024-10-01 23:36:03 +02:00
Ivan FB
696587fdac chore: Optimize hash queries with lookup table (#2933)
* Upgrade Postgres schema to add messages_lookup table
* Perform optimized query for messageHash-only queries
2024-08-08 21:46:08 +02:00
Simon-Pierre Vivier
d60ff3e0e6 chore(archive): archive and drivers refactor (#2761)
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
  The id column is needed because it contains the message digest
  which is used in store v2, and we need to keep support to
  store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
  We want the nwaku node to simultaneously support store-v2
  requests and store-v3 requests.
  Only the legacy archive is in charge of archiving messages, and the
  archived information is suitable to fulfill both store-v2 and
  store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2024-07-12 18:19:12 +02:00
Darshan K
8da2a9c0a5 chore: refactor relative path to better absolute (#2861) 2024-07-06 00:03:38 +02:00
Ivan FB
75ed5d14e3 chore: archive.nim - increase the max limit of content topics per query to 100 (#2846) 2024-06-26 12:24:15 +02:00
Ivan FB
21a96cea7d postgres partitions: ensure the partition ranges are o'clock (#2776)
Also, skip the error "partition already exists" because that happens
when multiple nodes interact with the same database.
2024-06-05 17:45:38 +02:00
Simon-Pierre Vivier
6ab1084d46 fix: invalid cursor returning messages (#2724)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-27 10:54:10 -04:00
Simon-Pierre Vivier
2db1cf3fcb fix: store v3 bug fixes (#2718) 2024-05-23 08:01:52 -04:00
Álex Cabeza Romero
9dd59c727e test(peer-and-connection-management): Implement tests (#2566)
* Implement peer and connection management tests.
* Fix multiple peers added on initialisation.
* Remove clusterId parameter from newTestWakuNode.
2024-05-13 17:25:44 +02:00
Ivan FB
5440dd3fbc chore: Postgres enhance get oldest timestamp (#2687)
* postgres: consider also the existing paritions when getting oldest timestamp
* test_driver_postgres_query: adapt test to oldest timestamp
2024-05-10 18:31:01 +02:00
gabrielmer
bbffa88a0d fix: use await instead of waitFor in async tests (#2690) 2024-05-10 14:13:58 +02:00
gabrielmer
9be3221b5d feat: supporting meta field in store (#2609) 2024-05-06 10:20:21 +02:00
Simon-Pierre Vivier
db72e2b823 fix: store v3 validate cursor & remove messages (#2636) 2024-05-01 14:47:06 -04:00
Simon-Pierre Vivier
665d9e3a06 feat: store v3 (#2431) 2024-04-25 09:09:52 -04:00
Ivan FB
588530d5c7 chore: Better postgres duplicate insert (#2535)
* postgres_driver: add ON CONFLICT DO NOTHING in the insert statement
* test_driver_postgres: adapt test to ON CONFLICT DO NOTHING
  The insert does not fail when duplicate, it returns a positive response
  when doing 'put' of a duplicated row. The test is adapted so that
  we just check that the number of messages doesn't grow after
  trying to add a duplicated row.
2024-03-18 15:59:45 +01:00
Ivan FB
cf6298ca1f Generic re-style with nph 0.5.1 (#2396) 2024-03-16 00:08:47 +01:00
Simon-Pierre Vivier
430708ccc6 feat: archive update for store v3 (#2451) 2024-03-12 07:51:03 -04:00
Ivan FB
132bb0bbf2 feat: Postgres partition implementation (#2506)
* postgres: first step to implement partition management
* postgres_driver: use of times.now().toTime().toUnix() instead of Moment.now()
* postgres migrations: set new version to 2
* test_driver_postgres: use of assert instead of require and avoid using times.now()
* postgres_driver: better implementation of the reset method with partitions
* Remove createMessageTable, init, and deleteMessageTable procs
* postgres: ensure we use the version 15.4 in tests
* postgres_driver.nim: enhance debug logs partition addition
* ci.yml: ensure logs are printed without colors
* postgres_driver: starting the loop factory in an asynchronous task
* postgres_driver: log partition name and size when removing a partition
2024-03-06 20:50:22 +01:00
Ivan FB
75521122a4 chore: Postgres migrations (#2477)
* Add postgres_driver/migrations.nim
* Postgres and archive logic adaptation to the migration implementation
* libwaku: adapt node_lifecycle_request.nim to migration refactoring
* test_app.nim: add more detail for test that only fails in CI
* postgres migrations: store the migration scripts inside the resulting wakunode binary instead of external .sql files.
2024-03-01 12:05:27 +01:00
Ivan FB
0887a344a4 refactor: new proc to foster different size retention policy implementations (#2463)
* new proc to foster different size retention policy implementations
  The new proc, decreaseDatabaseSize, will have different implementations
  per each driver. For example, in future commits we will implement a size
  retention policy thanks to partitions management, in Postgres.
* RetentionPolicy: use of new instead of init for ref object types
* waku_archive: fix signatures in decreaseDatabaseSize methods
* retention_policy_size: minor cleanup of comments and imports
2024-02-22 16:55:37 +01:00
Álex Cabeza Romero
c3dea59e8f test(lightpush): Lightpush functional tests (#2269)
* Add ligthpush payload tests.
* Add end to end lightpush tests.
* updating vendor/nim-unittest2 to protect against core dump issue
* Enable "Valid Payload Sizes" test again
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-02-06 17:37:42 +01:00
Álex Cabeza Romero
91c402b1cc test(store): Implement store tests (#2235, 2240)
* Implement store tests
2023-12-19 15:38:43 +01:00
Álex Cabeza Romero
b8b32eb857 refactor(store): HistoryQuery.direction (#2263)
* Fix issue with default history query ascending value in serde operations: Should use the same value.
* Update direction types to PagingDirection.
2023-12-19 15:10:27 +01:00
Ivan FB
88f2a9f89b test_driver_postgres: enhance test coverage, multiple and single topic (#2301)
Co-authored-by: Abhimanyu <ABresting@users.noreply.github.com>
2023-12-19 10:41:50 +01:00
Ivan FB
32d5c24d29 fix: add protection in rest service to always publish with timestamp if user doesn't provide it (#2261) 2023-12-06 14:02:21 +01:00
Álex Cabeza Romero
a5a85981c5 chore(store-archive): Remove duplicated code (#2234)
* Refactor utility functions for store and archive test.
2023-11-27 18:33:27 +01:00
Abhimanyu
d840d3e7cf fix: extended Postgres code to support retention policy + refactoring (#2244)
* updated Postgres retention policy code + refactoring

* Update waku/waku_archive/driver/postgres_driver/postgres_driver.nim

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>

* updated code review changes

* data unit fixed, processing everything in bytes now

---------

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>
2023-11-24 15:43:47 +01:00
Abhimanyu
60ed32c25a feat: Add new DB column messageHash (#2202)
* feat: added DB column messageHash

* feat: minor change

* feat: minor merge conflict fix

* Update test_resume.nim

* Update test_resume.nim

* randomblob() func used to populate attribute

* PRIMARY key updated - SQLite and Postgres
2023-11-22 17:32:56 +01:00
Abhimanyu
9f9c83e984 chore: removing automatic vacuuming from retention policy code (#2228)
* retention policy and testcase updated

* removing dead code

* review updated code
2023-11-21 11:27:50 +01:00
Abhimanyu
06a5acb6e9 Revert "feat: amending computeDigest func. + related test cases (#2132)" (#2180)
This reverts commit 8ea31ac6439498de3ab1b6641633a781bf1d64bd.
2023-11-08 01:41:23 +01:00
Ivan FB
4759388664 chore: Optimize postgres - prepared statements in select (#2182)
* db_postgres: use prepared statements on most freq select queries
* db_postgres/dbconn.nim adding better feedback in case of query error
* dbconn: use of isOkOr
* pgasyncpool: refactor to reduce code (valueOr, catch:)
2023-11-07 13:38:37 +01:00
Abhimanyu
69ddeb93bd Revert "feat: messageHash attribute added in SQLite + testcase (#2142)" (#2154)
This reverts commit a49a0e3c5b475f8a14ad7298e3d1b485969f01d8.
2023-10-24 16:05:39 +02:00
Abhimanyu
a49a0e3c5b feat: messageHash attribute added in SQLite + testcase (#2142)
* feat: messageHash attaribute added in SQLite + testcase

* Update tests/waku_archive/test_driver_sqlite_query.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2023-10-24 12:19:52 +02:00
Abhimanyu
8ea31ac643 feat: amending computeDigest func. + related test cases (#2132)
* feat: amending computeDigest func. + related test cases

* minor fixes

* minor fixes v1: testcase saga continues

---------

Co-authored-by: Vaclav Pavlin <vaclav@status.im>
2023-10-19 11:59:17 +02:00
Abhimanyu
fb24830fcc chore: added size based retention policy (#2098)
* chore: added size based retention policy

* chore: post review code - size based retention policy

* chore: review integrated, size retention updated, vacuuming in capacity retention

* chore: typo fixed

* chore: review integrated

* chore: external config updated to support newly added retention policy
2023-10-10 15:29:09 +05:30
Hanno Cornelius
8e644bf1ca Revert "chore: add size retention policy (#2093)" (#2097)
This reverts commit fc810eaf638af444c4e2002a21305fd78bca2d53.

(cherry picked from commit b213b2c385b0534481448cd6e30af18e183d0504)
2023-10-02 11:07:13 +02:00
Abhimanyu
fc810eaf63 chore: add size retention policy (#2093)
* chore: add retention policy with GB or MB limitation #1885

* chore: add retention policy with GB or MB limitation

* chore: updated code post review- retention policy

* ci: extract discordNotify to separate file

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: push images to new wakuorg/nwaku repo

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: enforce default Docker image tags strictly

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* ci: push GIT_REF if it looks like a version

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* fix: update wakuv2 fleet DNS discovery enrtree

https://github.com/status-im/infra-misc/issues/171

* chore: resolving DNS IP and publishing it when no extIp is provided (#2030)

* feat(coverage): Add simple coverage (#2067)

* Add test aggregator to all directories.
* Implement coverage script.

* fix(ci): fix name of discord notify method

Also use absolute path to load Groovy script.

Signed-off-by: Jakub Sokołowski <jakub@status.im>

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version (#2080)

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version

* add more metrics, refactor how most metrics are calculated

* rework metrics table fillup

* reset connErr to make sure we honour successful reconnection

* chore(cbindings): Adding cpp example that integrates the 'libwaku' (#2079)

* Adding cpp example that integrates the `libwaku`

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>

* fix(ci): update the dependency list in pre-release WF (#2088)

* chore: adding NetConfig test suite (#2091)

---------

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Anton Iakimov <yakimant@gmail.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>
Co-authored-by: Vaclav Pavlin <vaclav@status.im>
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2023-09-30 11:10:52 +05:30
Álex Cabeza Romero
67a872b291 feat(coverage): Add simple coverage (#2067)
* Add test aggregator to all directories.
* Implement coverage script.
2023-09-27 15:53:23 +02:00
Hanno Cornelius
5e2e1b6bd0 chore: remove references to v2 (#1898)
* chore: remove references to v2

* fix: lingering rln-relay import path
2023-08-09 18:11:50 +01:00