1189 Commits

Author SHA1 Message Date
Ivan FB
0fc617ff69
bug fix: update num-msgs archive metrics every minute and not only at the beginning (#2287) 2023-12-14 17:00:13 +01:00
Ivan FB
a0ee3d78fe
postgres_driver.nim: restrict getMessages prepared stmt to query with 1 content topic (#2296)
Before this commit, the following execution of a prepared statement
returned nothing even though the database had 2 rows to be returned:

nwaku-db-1  | 2023-12-14 12:55:17.575 UTC [73] LOG:  execute SelectWithoutCursorAsc: SELECT storedAt, contentTopic, payload, pubsubTopic, version, timestamp, id FROM messages
nwaku-db-1  | 	    WHERE contentTopic IN ($1) AND
nwaku-db-1  | 	          pubsubTopic = $2 AND
nwaku-db-1  | 	          storedAt >= $3 AND
nwaku-db-1  | 	          storedAt <= $4
nwaku-db-1  | 	    ORDER BY storedAt ASC LIMIT $5;
nwaku-db-1  | 2023-12-14 12:55:17.575 UTC [73] DETAIL:  parameters: $1 =
'my/ctopic/1,my/ctopic/2', $2 = '/waku/2/default-waku/proto', $3 = '1702552968570786800', $4 = '1702552968585347557', $5 = '101'

The reason why it is not returning anything is that the 'IN' statement doesn't work when using prepared statements with multiple items. It only works when the 'IN' content, i.e. $1, contains one single item.
2023-12-14 15:37:12 +01:00
Ivan FB
385daf16be
chore: vendor bump for 0.23.0 (#2274)
* on_chain/group_manager: use .async: (raises:[Exception]).
* bump nim-dnsdisc
* update nim-chronos to the latest state
* chat2.nim: catch any possible exception when stopping
* chat2bridge.nim: make it to compile after vendor bump
* ValidIpAddress (deprecated) -> IpAddress
* vendor/nim-libp2p additional bump
* libwaku: adapt to vendor bump
* testlib/wakunode.nim: adapt to vendor bump (ValidIpAddress -> IpAddress)
* waku_node: avoid throwing any exception from stop*(node: WakuNode)
* test_confutils_envvar.nim: ValidIpAddress -> IpAddress
* test_jsonrpc_store: capture exception
* test_rln*: handling exceptions
* adaptation to make test_rln_* to work properly
* signature enhancement of group_manager methods
2023-12-14 07:16:39 +01:00
Ivan FB
0cc0c80563
chore: peer_manager.nim - reduce logs from debug to trace (#2279) 2023-12-12 16:00:18 +01:00
Aaryamann Challani
15c1f974df
fix(rln-relay): graceful retries on rpc calls (#2250)
* fix(rln-relay): graceful retries on rpc calls

* fix: missing file
2023-12-11 14:59:16 +05:30
Ivan FB
77c5ba7669
archive: simplify and enhance async retention policy application (#2278)
* Avoid using timer and just use an infinite async loop that can be
cancelled at any time.
2023-12-11 08:50:40 +01:00
Simon-Pierre Vivier
281c13a429
added sharded peer store pruning (#2167) 2023-12-07 07:21:18 -05:00
Simon-Pierre Vivier
dba9820c1f
feat: shard aware peer management (#2151) 2023-12-07 06:48:28 -05:00
Ivan FB
42f1957920
fix: add protection in rest service to always publish with timestamp if user doesn't provide it (#2261) 2023-12-06 14:02:21 +01:00
Sasha
103d3981ad
fix: remove trailing commas from keystore json (#2200)
* fix: remove trailing commas from keystore json

* keyfile.nim: try a different Json formatting approach

* build keystore

* address comment

---------

Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2023-12-01 12:57:19 +01:00
Ivan FB
89dbbd2048
waku_store/common.nim: correct ret code in PEER_DIAL_FAILURE (#2260) 2023-11-30 16:42:58 +01:00
Ivan FB
72a1f8c724
chore: Better feedback invalid content topic (#2254)
* typo correction appplication -> application
* content_topic.nim: better feedback to user when wrong topic is passed
* test_namespaced_topics.nim: updating tests accordingly
2023-11-30 11:11:33 +01:00
omahs
958b9bd712
chore: fix typos (#2239) 2023-11-30 11:08:08 +01:00
Ivan FB
6301bec05d
fix: waku_filter_v2/common: PEER_DIAL_FAILURE ret code change: 200 -> 504 (#2236) 2023-11-30 10:47:45 +01:00
Simon-Pierre Vivier
bebaa59c3b
chore(rest): refactor message cache (#2221) 2023-11-28 07:21:41 -05:00
Simon-Pierre Vivier
c301e880c8
chore: refactoring peer storage (#2243) 2023-11-27 08:08:58 -05:00
Ivan FB
b31c182325
chore: postres driver allow setting the max number of connection from a parameter (#2246)
* postres driver: allow setting the max number of connections from a parameter
2023-11-24 16:21:22 +01:00
Abhimanyu
a1ed517f9c
fix: extended Postgres code to support retention policy + refactoring (#2244)
* updated Postgres retention policy code + refactoring

* Update waku/waku_archive/driver/postgres_driver/postgres_driver.nim

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>

* updated code review changes

* data unit fixed, processing everything in bytes now

---------

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>
2023-11-24 15:43:47 +01:00
Abhimanyu
aeb77a3e75
feat: Add new DB column messageHash (#2202)
* feat: added DB column messageHash

* feat: minor change

* feat: minor merge conflict fix

* Update test_resume.nim

* Update test_resume.nim

* randomblob() func used to populate attribute

* PRIMARY key updated - SQLite and Postgres
2023-11-22 17:32:56 +01:00
Abhimanyu
a22ee60494
chore: deterministic message hash algorithm updated (#2233)
* deterministic hash algorithm updated + testcases

* updated code review
2023-11-22 15:23:43 +01:00
gabrielmer
59ee3c69c8
chore(REST): returning lightpush support and updated filter protocol (#2219) 2023-11-22 10:56:23 +02:00
Ivan FB
c48accb513
waku_store: better response when the store is requested with wrong cursor (#2231) 2023-11-22 09:32:39 +01:00
Simon-Pierre Vivier
897f487978
chore: mics. improvements to cluster id and shards setup (#2187) 2023-11-21 15:15:39 -05:00
Alvaro Revuelta
51f36099d5
fix(rln): error in api when rate limit (#2212) 2023-11-21 19:24:31 +01:00
Ivan FB
c973b85069
peer_manager.nim: better feedback if can't dial peer with WakuMetadataCodec (#2230) 2023-11-21 14:54:45 +01:00
Abhimanyu
9ff441ab47
chore: removing automatic vacuuming from retention policy code (#2228)
* retention policy and testcase updated

* removing dead code

* review updated code
2023-11-21 11:27:50 +01:00
Ivan FB
13fb7c2f74
group_manager.nim more except detail when cant connect eth client (#2195) 2023-11-20 23:25:55 +01:00
gabrielmer
ef8ffbdbd0
chore: decoupling announced and listen addresses (#2203) 2023-11-16 18:15:27 +02:00
Álex Cabeza Romero
f5f431382b
test(waku-relay): Relay (#2101)
* Implement message id tests.
* Implement relay tests.
* Update import paths to use test_all.
2023-11-15 16:11:36 +01:00
Álex Cabeza Romero
85265e6c61
test(waku-filter): Unsubscribe tests (#2085)
* Implement unsubscribe waku filter tests.
* test(waku-filter): Unsubscribe all, payloads and security tests (#2095)
* Implement waku node filter Security and Privacy tests (#2096)
2023-11-15 10:26:01 +01:00
NagyZoltanPeter
6d81e3841a
chore: Allow text/plain content type descriptor for json formatted content body (#2209)
* Allow text/plain content type descriptor for json formatted content body. Refactored duplicated encode/decode functions for rest api

* Fix relay endpoint decodings of content bodies to accept text/plain

* Added support for content body decoder for checking media type if additional parameters are present

* Fix wrong usage of ContentTypeData - appeared only for tests
2023-11-14 16:59:53 +01:00
Vaclav Pavlin
2cb0989a28
fix(rest): properly check if rln is used (#2205)
* fix(rest): properly check if rln is used

* fix(apis): fix remaining usage of defined(rln)
2023-11-10 15:25:07 +01:00
Abhimanyu
d7ef3ca192
Revert "feat: amending computeDigest func. + related test cases (#2132)" (#2180)
This reverts commit 1669f710ce8f48d1b7fb06ca98433f2b25a44e6d.
2023-11-08 01:41:23 +01:00
Ivan FB
6da1aeec53
chore: Optimize postgres - prepared statements in select (#2182)
* db_postgres: use prepared statements on most freq select queries
* db_postgres/dbconn.nim adding better feedback in case of query error
* dbconn: use of isOkOr
* pgasyncpool: refactor to reduce code (valueOr, catch:)
2023-11-07 13:38:37 +01:00
Simon-Pierre Vivier
200a11da09
feat(discv5): filter out peers without any listed capability (#2186) 2023-11-06 07:31:36 -05:00
gabrielmer
b9563ae0b5
chore: upgrade dependencies v0.22 (#2185) 2023-11-06 13:30:34 +02:00
Ivan FB
fa467e2462
fix: lightpush rest (#2176)
* rest/lightpush/handlers.nim: enhance feedback in case of error.
* lightpush/openapi.yaml: fix typo in pubsubTopic field.
2023-11-01 11:30:53 +01:00
Ivan FB
2b4ca4d0ff
chore: Optimize postgres - use of rowCallback approach (#2171)
* db_postgres, postgres_driver: better performance by using callback.
  There were a bunch of milliseconds being lost due to multiple-row
  processing. This commit aims to have the minimum possible row
  process time.
* pgasyncpool: clarifying logic around pool conn management.
* db_postgres: removing duplicate code and more searchable proc names.
2023-10-31 14:46:46 +01:00
Simon-Pierre Vivier
bcf8e9630d
feat: metadata protocol shard subscription (#2149) 2023-10-30 16:58:15 -04:00
Alvaro Revuelta
250e8b983c
fix(rest): fix bug in rest api when sending rln message (#2169) 2023-10-30 16:19:49 +01:00
Alvaro Revuelta
f0f69b3235
chore(networking): lower dhigh to limit amplification factor (#2168) 2023-10-30 16:17:39 +01:00
Ivan FB
282c2e8107
chore: Minor Postgres optimizations (#2166)
* postgres_healthcheck: validate once per minute instead of 30 sec
* postgres_driver.nim: change MaxNumCons from 5 to 50
* postgres_driver.nim: split connPool into writeConPool and readConPool
  This aims to avoid clashes in insert and select queries
  because the inserts and selects can happen concurrently
  in relay and store events, respectively.
2023-10-30 15:16:49 +01:00
Simon-Pierre Vivier
7ca516a5f8
feat: REST APIs discovery handlers (#2109) 2023-10-27 15:43:54 -04:00
NagyZoltanPeter
b8bcb1e74b
Pull new version of nim-presto that implements RestServer' new error handler callback (#2144)
Added rest request error handler to capture calls on not installed endpoints
better, more descriptive error message returned.
2023-10-27 16:31:57 +02:00
gabrielmer
f7b9afc26f
feat: implementing port 0 support (#2125) 2023-10-27 10:11:47 +03:00
gabrielmer
944dfdaa99
chore: adding ext-multiaddr-only CLI flag (#2141) 2023-10-24 18:39:25 +03:00
Abhimanyu
13aeebe46f
Revert "feat: messageHash attribute added in SQLite + testcase (#2142)" (#2154)
This reverts commit 9cd8c73d273bdfc95e6bcf870cd94030690f2630.
2023-10-24 16:05:39 +02:00
Abhimanyu
9cd8c73d27
feat: messageHash attribute added in SQLite + testcase (#2142)
* feat: messageHash attaribute added in SQLite + testcase

* Update tests/waku_archive/test_driver_sqlite_query.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2023-10-24 12:19:52 +02:00
Abhimanyu
1669f710ce
feat: amending computeDigest func. + related test cases (#2132)
* feat: amending computeDigest func. + related test cases

* minor fixes

* minor fixes v1: testcase saga continues

---------

Co-authored-by: Vaclav Pavlin <vaclav@status.im>
2023-10-19 11:59:17 +02:00
gabrielmer
a5b1cfd024
fix: consider WS extMultiAddrs before publishing host address (#2122) 2023-10-17 12:53:41 +03:00