86 Commits

Author SHA1 Message Date
Ivan Folgueira Bande
2b2df28544
fix compilation issues 2024-10-25 10:47:33 +02:00
Ivan Folgueira Bande
85f5e1b469
archive: continue giving some more clarity in error logs 2024-10-24 19:09:24 +02:00
Simon-Pierre Vivier
80c7581ab1
fix: add log and archive message ingress for sync (#3133) 2024-10-23 07:25:07 -04:00
richΛrd
c35dc54923
fix: add a limit of max 10 content topics per query (#3117) 2024-10-16 17:55:04 -04:00
richΛrd
c861fa9f75
chore: remove warnings (#3106)
- Removes deprecation and unused import warnings for libwaku
- Removes unused imports
- Adds .base. pragma to `SubscriptionObserver.onSubscribe`
- Uses casting for uint to enums conversions
- Bumps nim-chronicles
2024-10-10 08:40:09 -04:00
Ivan FB
5875ed63af
chore: Optimize store (#3061)
* use messages_lookup to retrieve timestamps
* deep refactoring in db_postgres for better use of async approach
2024-10-01 23:36:03 +02:00
Ivan FB
d68b06f145
chore: Better database query logs and logarithmic scale in grafana store panels (#3048) 2024-09-20 17:43:56 +02:00
fryorcraken
ce9a8c468a
chore: use submodule nph in CI to check lint (#3027) 2024-09-11 11:51:42 +10:00
Simon-Pierre Vivier
e1e05afb02
chore: per limit split of PostgreSQL queries (#3008) 2024-09-04 10:17:28 -04:00
Ivan FB
e8a49b76b2
chore: Better timing and requestId detail for slower store db queries (#2994)
* Better timing and requestId detail for store db queries slower than two seconds
* Adapt tests and client to allow sending custom store requestId
2024-08-29 22:56:14 +02:00
Ivan FB
32f2d85dcc
chore: ANALYZE messages query should be performed regularly (#2986)
---------

Co-authored-by: Richard Ramos <info@richardramos.me>
2024-08-21 19:17:08 +02:00
Ivan FB
b8550c5550
postgres_driver limit max num hashes to 100 (#2976) 2024-08-19 11:12:31 +02:00
Ivan FB
71d4a11647
faster retention policy when error and use of detach finalize when needed (#2966) 2024-08-12 10:47:01 +02:00
richΛrd
5f0fbd78c4
fix: return on insert error (#2956) 2024-08-11 21:35:04 -04:00
Ivan FB
4d47aa655f
chore: Optimize hash queries with lookup table (#2933)
* Upgrade Postgres schema to add messages_lookup table
* Perform optimized query for messageHash-only queries
2024-08-08 21:46:08 +02:00
Ivan FB
71ee42de56
chore: simple PR to enhance postgres and retention policy logs (#2884) 2024-07-15 20:58:31 +02:00
Ivan FB
241fb8cb97
chore: Update master from release v0.30 (#2908)
* chore(rln): rln message limit to 100 (#2883)
* postgres_driver: add more error handling when creating partitions
   Given that multiple nodes can be connected to the same database,
   it can happen that other node did something that my node was willing
   to do. In this commit, we overcome the possible "interleaved" 
   partition creation.

---------

Co-authored-by: Alvaro Revuelta <alvrevuelta@gmail.com>
2024-07-15 18:00:44 +02:00
Simon-Pierre Vivier
f54ba10bc7
chore(archive): archive and drivers refactor (#2761)
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
  The id column is needed because it contains the message digest
  which is used in store v2, and we need to keep support to
  store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
  We want the nwaku node to simultaneously support store-v2
  requests and store-v3 requests.
  Only the legacy archive is in charge of archiving messages, and the
  archived information is suitable to fulfill both store-v2 and
  store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2024-07-12 18:19:12 +02:00
Ivan FB
4ac4ab2a41
chore: sqlite make sure code is always run (#2891) 2024-07-10 18:34:28 +02:00
Ivan FB
9576b77c76
postgres_driver: better partition creation without exclusive access (#2887) 2024-07-09 13:41:29 +02:00
Ivan FB
fd6a71cdd7
chore: Bump dependencies for v0.31.0 (#2885)
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
  Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
  not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
2024-07-09 13:14:28 +02:00
Darshan K
9bd8c33aee
chore: remove all pre-nim-1.6 deadcode from codebase (#2857) 2024-06-28 16:04:57 +05:30
Ivan FB
a05fa06919
chore: archive.nim - increase the max limit of content topics per query to 100 (#2846) 2024-06-26 12:24:15 +02:00
Ivan FB
ab4b5b27bd
postgres_driver: better sync lock in partition creation (#2809)
With the original approach it happened cases where one connection
acquired the lock and other connection tried to release the same lock,
causing an unrecoverable failure which made the node to end.
2024-06-14 10:07:41 +02:00
Ivan FB
f5d87c5b23
chore: set msg_hash logs to notice level (#2737) 2024-06-10 15:56:55 +02:00
Ivan FB
31bd6d71c0
chore: Minor enhancements (#2789)
* archive.nim: reduce the database report interval from 1 to 30 min
  This counts the number of rows with "select count(1) from messages"
  which is quite intense and we shouldn't run it every minute
* aside cleanup
2024-06-09 23:09:23 +02:00
Ivan FB
c5d19c4491
chore: postgres_driver - acquire/release advisory lock when creating partitions (#2784) 2024-06-07 17:54:26 +02:00
Ivan FB
dbe79250da
postgres_driver: simple reformat with nph (#2785) 2024-06-06 12:04:40 +02:00
Ivan FB
3a93437300
add new index to optimize the order by storedAt (#2778) 2024-06-06 11:38:58 +02:00
Ivan FB
24661ea969
postgres partitions: ensure the partition ranges are o'clock (#2776)
Also, skip the error "partition already exists" because that happens
when multiple nodes interact with the same database.
2024-06-05 17:45:38 +02:00
Darshan K
dfc979a8ef
chore: extract common prefixes into a constant for multiple query (#2747)
* chore: extract select to constant for multiple prefix

* chore: Add a space at the end of selectClause to facilitate better string merging.

Signed-off-by: DarshanBPatel <darshan@status.im>

---------

Signed-off-by: DarshanBPatel <darshan@status.im>
2024-06-03 21:52:53 +05:30
richΛrd
88b8e1867a
fix(waku_archive): only allow a single instance to execute migrations (#2736) 2024-05-31 12:08:16 -04:00
Ivan FB
8b42f199ba
postgres_driver.nim: add missing meta field to select queries (#2741) 2024-05-29 22:13:16 +02:00
Simon-Pierre Vivier
a65b13fcb8
fix: invalid cursor returning messages (#2724)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-27 10:54:10 -04:00
Simon-Pierre Vivier
4a6ec468db
fix: store v3 bug fixes (#2718) 2024-05-23 08:01:52 -04:00
Ivan FB
78132dc12e
chore: generic change to reduce the number of compilation warnings (#2696) 2024-05-16 22:29:11 +02:00
Ivan FB
8451cf8e1b
chore: Postgres enhance get oldest timestamp (#2687)
* postgres: consider also the existing paritions when getting oldest timestamp
* test_driver_postgres_query: adapt test to oldest timestamp
2024-05-10 18:31:01 +02:00
Simon-Pierre Vivier
d700006a6a
feat: store v3 return pubsub topics (#2676) 2024-05-08 15:35:56 -04:00
Ivan FB
b46226fca0
postgres_driver: delete partitions in time retention policy (#2679) 2024-05-07 23:42:01 +02:00
richΛrd
82f95999cd
fix: add meta to sqlite migration scripts (#2675) 2024-05-07 09:39:06 -04:00
gabrielmer
a46d4451eb
feat: supporting meta field in store (#2609) 2024-05-06 10:20:21 +02:00
Ivan FB
f8184a6de8
postgres_driver.nim: debug -> trace put PotsgresDriver (#2667) 2024-05-03 17:41:14 +02:00
Simon-Pierre Vivier
e03d1165e6
fix: store v3 validate cursor & remove messages (#2636) 2024-05-01 14:47:06 -04:00
Ivan FB
d5e0e4a9b1
chore: log enhancement for message reliability analysis (#2640)
* log enhancement for message reliability analysis

The next modules are touched:
  - waku_node.nim
  - archive.nim
  - waku_filter_v2/protocol.nim
  - waku_relay/protocol.nim

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
2024-05-01 10:25:33 +02:00
Simon-Pierre Vivier
0b0fbfad5c
feat: store v3 (#2431) 2024-04-25 09:09:52 -04:00
gabrielmer
4117fe65b7
chore: adding migration script adding i_query index (#2578) 2024-04-15 12:57:35 +03:00
Ivan FB
b5e2edb724
postgres_driver.nim: make sure the partition list ist loaded in correct order (#2548) 2024-03-25 18:02:07 +01:00
Ivan FB
693a1778d1
chore: Better postgres duplicate insert (#2535)
* postgres_driver: add ON CONFLICT DO NOTHING in the insert statement
* test_driver_postgres: adapt test to ON CONFLICT DO NOTHING
  The insert does not fail when duplicate, it returns a positive response
  when doing 'put' of a duplicated row. The test is adapted so that
  we just check that the number of messages doesn't grow after
  trying to add a duplicated row.
2024-03-18 15:59:45 +01:00
Ivan FB
3ba4378cbe
Generic re-style with nph 0.5.1 (#2396) 2024-03-16 00:08:47 +01:00
richΛrd
23a291b372
fix(store): retention policy regex (#2532) 2024-03-15 09:46:35 -04:00