75 Commits

Author SHA1 Message Date
Ivan FB
454b098ac5
new metric in postgres_driver to estimate payload stats (#3596) 2025-11-24 10:16:37 +01:00
Fabiana Cecin
7b580dbf39
chore(refactoring): replace some isErr usage with better alternatives (#3615)
* Closes apply isOkOr || valueOr approach (#1969)
2025-10-27 14:07:06 -03:00
Ivan FB
7e5041d5e1
Move log level from debug to info (#3622)
* convert all debug logs to info log level
* waku_relay protocol mv notice spammy logs to debug
2025-10-15 10:49:36 +02:00
Fabiana Cecin
4d68e2abd5
chore(refactoring): results lib refactors (mostly replace isOk) (#3610)
* Changes isOk usage into better patterns with e.g. valueOr / isOkOr
* Some other refactoring included
* This PR partially addresses #1969
2025-10-08 19:14:54 -03:00
Ivan FB
2936ba838d fix: detach partition (#3535)
* fix to make sure partitions get properly detached
2025-08-24 22:58:06 +02:00
Miran
ed0474ade3 chore: fix unused and deprecated imports (#3368) 2025-04-11 18:20:23 +03:00
gabrielmer
4d9e11f16b chore: adding extra migration to sqlite and improving error message (#3240) 2025-01-16 16:10:28 +01:00
Ivan FB
9d0b30cc99 chore: golang example end using negentropy dependency plus simple readme.md (#3235)
* avoid dependency with libpcre by using regex module
2025-01-15 10:32:22 +01:00
Ivan FB
e128385e69 chore: Optimize store (#3061)
* use messages_lookup to retrieve timestamps
* deep refactoring in db_postgres for better use of async approach
2024-10-01 23:36:03 +02:00
Ivan FB
96cc2f1b39 chore: Better database query logs and logarithmic scale in grafana store panels (#3048) 2024-09-20 17:43:56 +02:00
fryorcraken
723b009b20 chore: use submodule nph in CI to check lint (#3027) 2024-09-11 11:51:42 +10:00
Simon-Pierre Vivier
ed866321a0 chore: per limit split of PostgreSQL queries (#3008) 2024-09-04 10:17:28 -04:00
Ivan FB
236547ec7d chore: Better timing and requestId detail for slower store db queries (#2994)
* Better timing and requestId detail for store db queries slower than two seconds
* Adapt tests and client to allow sending custom store requestId
2024-08-29 22:56:14 +02:00
Ivan FB
2e8f2f0076 chore: ANALYZE messages query should be performed regularly (#2986)
---------

Co-authored-by: Richard Ramos <info@richardramos.me>
2024-08-21 19:17:08 +02:00
Ivan FB
73fbe5c337 postgres_driver limit max num hashes to 100 (#2976) 2024-08-19 11:12:31 +02:00
Ivan FB
6ae46c5fff faster retention policy when error and use of detach finalize when needed (#2966) 2024-08-12 10:47:01 +02:00
Ivan FB
696587fdac chore: Optimize hash queries with lookup table (#2933)
* Upgrade Postgres schema to add messages_lookup table
* Perform optimized query for messageHash-only queries
2024-08-08 21:46:08 +02:00
Ivan FB
b196bc097b chore: simple PR to enhance postgres and retention policy logs (#2884) 2024-07-15 20:58:31 +02:00
Ivan FB
d6e53631e5 chore: Update master from release v0.30 (#2908)
* chore(rln): rln message limit to 100 (#2883)
* postgres_driver: add more error handling when creating partitions
   Given that multiple nodes can be connected to the same database,
   it can happen that other node did something that my node was willing
   to do. In this commit, we overcome the possible "interleaved" 
   partition creation.

---------

Co-authored-by: Alvaro Revuelta <alvrevuelta@gmail.com>
2024-07-15 18:00:44 +02:00
Simon-Pierre Vivier
d60ff3e0e6 chore(archive): archive and drivers refactor (#2761)
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
  The id column is needed because it contains the message digest
  which is used in store v2, and we need to keep support to
  store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
  We want the nwaku node to simultaneously support store-v2
  requests and store-v3 requests.
  Only the legacy archive is in charge of archiving messages, and the
  archived information is suitable to fulfill both store-v2 and
  store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2024-07-12 18:19:12 +02:00
Ivan FB
9a9b773546 chore: sqlite make sure code is always run (#2891) 2024-07-10 18:34:28 +02:00
Ivan FB
36b4a6fff3 postgres_driver: better partition creation without exclusive access (#2887) 2024-07-09 13:41:29 +02:00
Ivan FB
13316201f7 chore: Bump dependencies for v0.31.0 (#2885)
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
  Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
  not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
2024-07-09 13:14:28 +02:00
Darshan K
7ad9722ecf chore: remove all pre-nim-1.6 deadcode from codebase (#2857) 2024-06-28 16:04:57 +05:30
Ivan FB
0bd3087de9 postgres_driver: better sync lock in partition creation (#2809)
With the original approach it happened cases where one connection
acquired the lock and other connection tried to release the same lock,
causing an unrecoverable failure which made the node to end.
2024-06-14 10:07:41 +02:00
Ivan FB
f958512e46 chore: Minor enhancements (#2789)
* archive.nim: reduce the database report interval from 1 to 30 min
  This counts the number of rows with "select count(1) from messages"
  which is quite intense and we shouldn't run it every minute
* aside cleanup
2024-06-09 23:09:23 +02:00
Ivan FB
4db31d2506 chore: postgres_driver - acquire/release advisory lock when creating partitions (#2784) 2024-06-07 17:54:26 +02:00
Ivan FB
69ce01d2be postgres_driver: simple reformat with nph (#2785) 2024-06-06 12:04:40 +02:00
Ivan FB
98aa45dc30 add new index to optimize the order by storedAt (#2778) 2024-06-06 11:38:58 +02:00
Ivan FB
21a96cea7d postgres partitions: ensure the partition ranges are o'clock (#2776)
Also, skip the error "partition already exists" because that happens
when multiple nodes interact with the same database.
2024-06-05 17:45:38 +02:00
Darshan K
7242d2aca2 chore: extract common prefixes into a constant for multiple query (#2747)
* chore: extract select to constant for multiple prefix

* chore: Add a space at the end of selectClause to facilitate better string merging.

Signed-off-by: DarshanBPatel <darshan@status.im>

---------

Signed-off-by: DarshanBPatel <darshan@status.im>
2024-06-03 21:52:53 +05:30
richΛrd
ae52913128 fix(waku_archive): only allow a single instance to execute migrations (#2736) 2024-05-31 12:08:16 -04:00
Ivan FB
e281a008fb postgres_driver.nim: add missing meta field to select queries (#2741) 2024-05-29 22:13:16 +02:00
Simon-Pierre Vivier
6ab1084d46 fix: invalid cursor returning messages (#2724)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-27 10:54:10 -04:00
Simon-Pierre Vivier
2db1cf3fcb fix: store v3 bug fixes (#2718) 2024-05-23 08:01:52 -04:00
Ivan FB
652fc172d4 chore: generic change to reduce the number of compilation warnings (#2696) 2024-05-16 22:29:11 +02:00
Ivan FB
5440dd3fbc chore: Postgres enhance get oldest timestamp (#2687)
* postgres: consider also the existing paritions when getting oldest timestamp
* test_driver_postgres_query: adapt test to oldest timestamp
2024-05-10 18:31:01 +02:00
Ivan FB
c3aa80284f postgres_driver: delete partitions in time retention policy (#2679) 2024-05-07 23:42:01 +02:00
richΛrd
7f7aa59c4b fix: add meta to sqlite migration scripts (#2675) 2024-05-07 09:39:06 -04:00
gabrielmer
9be3221b5d feat: supporting meta field in store (#2609) 2024-05-06 10:20:21 +02:00
Ivan FB
2a7984b951 postgres_driver.nim: debug -> trace put PotsgresDriver (#2667) 2024-05-03 17:41:14 +02:00
Simon-Pierre Vivier
db72e2b823 fix: store v3 validate cursor & remove messages (#2636) 2024-05-01 14:47:06 -04:00
Simon-Pierre Vivier
665d9e3a06 feat: store v3 (#2431) 2024-04-25 09:09:52 -04:00
gabrielmer
55d6b95287 chore: adding migration script adding i_query index (#2578) 2024-04-15 12:57:35 +03:00
Ivan FB
51bc92a18d postgres_driver.nim: make sure the partition list ist loaded in correct order (#2548) 2024-03-25 18:02:07 +01:00
Ivan FB
588530d5c7 chore: Better postgres duplicate insert (#2535)
* postgres_driver: add ON CONFLICT DO NOTHING in the insert statement
* test_driver_postgres: adapt test to ON CONFLICT DO NOTHING
  The insert does not fail when duplicate, it returns a positive response
  when doing 'put' of a duplicated row. The test is adapted so that
  we just check that the number of messages doesn't grow after
  trying to add a duplicated row.
2024-03-18 15:59:45 +01:00
Ivan FB
cf6298ca1f Generic re-style with nph 0.5.1 (#2396) 2024-03-16 00:08:47 +01:00
Simon-Pierre Vivier
430708ccc6 feat: archive update for store v3 (#2451) 2024-03-12 07:51:03 -04:00
Ivan FB
132bb0bbf2 feat: Postgres partition implementation (#2506)
* postgres: first step to implement partition management
* postgres_driver: use of times.now().toTime().toUnix() instead of Moment.now()
* postgres migrations: set new version to 2
* test_driver_postgres: use of assert instead of require and avoid using times.now()
* postgres_driver: better implementation of the reset method with partitions
* Remove createMessageTable, init, and deleteMessageTable procs
* postgres: ensure we use the version 15.4 in tests
* postgres_driver.nim: enhance debug logs partition addition
* ci.yml: ensure logs are printed without colors
* postgres_driver: starting the loop factory in an asynchronous task
* postgres_driver: log partition name and size when removing a partition
2024-03-06 20:50:22 +01:00
Aaryamann Challani
8b45204fda fix(postgres): import under feature flag (#2500) 2024-03-06 17:39:02 +05:30