1058 Commits

Author SHA1 Message Date
Prem Chaitanya Prathi
42204115c7
chore: log messageHash for lightpush request that helps in debugging (#2366) 2024-01-24 19:20:21 +05:30
Aaryamann Challani
efdc5244f6
chore(rln-relay): remove websocket from OnchainGroupManager (#2364)
* chore(rln-relay): remove websocket from OnchainGroupManager

* fix: swap ws for http
2024-01-23 23:22:45 +05:30
gabrielmer
f841454edc
chore: improve POST /relay/v1/auto/messages/{topic} error handling (#2339) 2024-01-18 13:49:13 +01:00
NagyZoltanPeter
c3358409bb
chore: Refactor of FilterV2 subscription management with Time-to-live maintenance (#2341)
* Refactor of FilterV2 subscription handling and maintenance with addition subscription time-to-live support.
Fixed all tests and reworked where subscription handling changes needed it.
Adapted REST API /admin filter subscription retrieve to new filter subscription structure.

* Fix tests and PR comments

* Added filter v2 subscription timeout tests and fixed

* Fix review comments and suggestions. No functional change.

* Remove leftover echoes from test_rest_admin

* Fix failed legacy filter tests due to separation of mounting the filters.

* Small fixes, fix naming typo, removed duplicated checks in test
2024-01-16 17:27:40 +01:00
gabrielmer
3d816c0814
chore: Bump nim-dnsdisc (#2354)
* chore: bump `vendor/nim-dnsdisc`

* chore: Update import path of dnsdisc

---------

Co-authored-by: Emil Ivanichkov <emil.ivanichkov@gmail.com>
2024-01-15 16:54:02 +02:00
gabrielmer
bcab9d55da
feat: adding filter data admin endpoint (REST) (#2314) 2024-01-09 11:42:29 +01:00
joao
3553fb4fc1
docs: Fix Typos Across Various Documentation Files (#2310) 2024-01-08 13:13:34 +01:00
Simon-Pierre Vivier
7b4f4bb18c
fix(discv5): add bootnode filter exception (#2267) 2024-01-04 16:39:03 -05:00
Ivan FB
bd7570faa3
bump vendors for 0.24.0 (#2333)
The following vendors have changes but are not being updated for
the reason explained.

nim-web3: not updated because unit tests started to fail and no
straightforward solution found.

nim-toml-serialization: not updated because it introduced a breaking
change on how the --config-file attribute is parsed. The array
attributes now need a comma. For example, the following attribute
from within the config file:

pubsub-topic = [ "/waku/2/default-waku/proto" "/waku/2/testing-store" ]

... should be converted to:

pubsub-topic = [ "/waku/2/default-waku/proto", "/waku/2/testing-store" ]

and we cannot accept that breaking change
2024-01-04 17:35:00 +01:00
Álex Cabeza Romero
3bbe234962
test(autosharding): Functional Tests (#2318)
* Implement autosharding tests.
2024-01-04 16:26:27 +01:00
Ivan FB
ed09074cc3
chore: message.nim - set max message size to 150KiB according to spec (#2298)
* message.nim: set max message size to 150KiB according to spec

Using KiB instead of KB because that seems more aligned with
the actual default defined in nim-libp2p (1024 * 1024)

Spec details: https://rfc.vac.dev/spec/64/#message-size

* test_protocol.nim: align test to current WakuMessage limit
* test_waku_client.nim: adapt test to MaxWakuMessageSize change
* make maxMessageSize configurable for wakunode2
* wakunode2 app now accepts max-num-bytes-msg-size with KiB, KB, or B units
* testlib/wakunode.nim: set maxMessageSize: "1024 KiB"
* test_waku_client.nim: remove duplicate check in "Valid Payload Sizes"
* set DefaultMaxWakuMessageSizeStr as the only source of truth
* external_config.nim: rename max-num-bytes-msg-size -> max-msg-size
2024-01-03 13:11:50 +01:00
Ivan FB
ebad0385ef
ip colocation is parameterizable. If set to 0, it is disabled (#2323)
The "ip colocation" concept refers to the maximum allowed peers
from the same IP address. For example, we allow disabling this limit when the
node works behind a reverse proxy.
2024-01-02 14:01:18 +01:00
Ivan FB
32668f43f5
fix: Revert "feat: shard aware peer management (#2151)" (#2312)
This reverts commit dba9820c1fa00f414f18d57f7a3ff38b67d2bb1a.

We need to revert this commit because
the waku-simulator stopped working. i.e. the nodes couldn't establish
connections among them: 054ba9e33f

Also, the following js-waku test fails due to this commit:
"same cluster, different shard: nodes connect"

* waku_lightpush/protocol.nim: minor changes to make it compile after revert
2023-12-20 15:23:41 +01:00
gabrielmer
a1b27edf80
fix: setting connectivity loop interval to 15 seconds (#2307) 2023-12-20 09:38:14 +01:00
Álex Cabeza Romero
fae20bff20
refactor(store): HistoryQuery.direction (#2263)
* Fix issue with default history query ascending value in serde operations: Should use the same value.
* Update direction types to PagingDirection.
2023-12-19 15:10:27 +01:00
Ivan FB
574cdf555a
chore: archive - move error to trace level when insert row fails (#2283)
* archive: move error to trace level when insert row fails

That is helpful to prevent the node to spam the logs when it shares
connection to the same Postgres database with other nodes, in
which case the following log appears too much:

topics="waku archive" tid=1 file=archive.nim:113 err="error in
runStmt: error in dbConnQueryPrepared calling waitQueryToFinish: error
in query: ERROR: duplicate key value violates unique constraint
"messageindex" DETAIL: Key
(messagehash)=(88f4ee115eef6f233a7dceaf975f03946e18666adda877e38d61be98add934e8)
already exists. "
2023-12-15 18:58:35 +01:00
gabrielmer
306c8a6259
chore: including content topics on FilterSubscribeRequest logs (#2295) 2023-12-15 14:18:12 +01:00
Ivan FB
c9d558f878
relay: add trace logs in case of msg validation rejection (#2285) 2023-12-15 13:34:30 +01:00
Alvaro Revuelta
5847f49d3d
fix: make rln rate limit spec compliant (#2294) 2023-12-15 10:26:17 +01:00
Ivan FB
0fc617ff69
bug fix: update num-msgs archive metrics every minute and not only at the beginning (#2287) 2023-12-14 17:00:13 +01:00
Ivan FB
a0ee3d78fe
postgres_driver.nim: restrict getMessages prepared stmt to query with 1 content topic (#2296)
Before this commit, the following execution of a prepared statement
returned nothing even though the database had 2 rows to be returned:

nwaku-db-1  | 2023-12-14 12:55:17.575 UTC [73] LOG:  execute SelectWithoutCursorAsc: SELECT storedAt, contentTopic, payload, pubsubTopic, version, timestamp, id FROM messages
nwaku-db-1  | 	    WHERE contentTopic IN ($1) AND
nwaku-db-1  | 	          pubsubTopic = $2 AND
nwaku-db-1  | 	          storedAt >= $3 AND
nwaku-db-1  | 	          storedAt <= $4
nwaku-db-1  | 	    ORDER BY storedAt ASC LIMIT $5;
nwaku-db-1  | 2023-12-14 12:55:17.575 UTC [73] DETAIL:  parameters: $1 =
'my/ctopic/1,my/ctopic/2', $2 = '/waku/2/default-waku/proto', $3 = '1702552968570786800', $4 = '1702552968585347557', $5 = '101'

The reason why it is not returning anything is that the 'IN' statement doesn't work when using prepared statements with multiple items. It only works when the 'IN' content, i.e. $1, contains one single item.
2023-12-14 15:37:12 +01:00
Ivan FB
385daf16be
chore: vendor bump for 0.23.0 (#2274)
* on_chain/group_manager: use .async: (raises:[Exception]).
* bump nim-dnsdisc
* update nim-chronos to the latest state
* chat2.nim: catch any possible exception when stopping
* chat2bridge.nim: make it to compile after vendor bump
* ValidIpAddress (deprecated) -> IpAddress
* vendor/nim-libp2p additional bump
* libwaku: adapt to vendor bump
* testlib/wakunode.nim: adapt to vendor bump (ValidIpAddress -> IpAddress)
* waku_node: avoid throwing any exception from stop*(node: WakuNode)
* test_confutils_envvar.nim: ValidIpAddress -> IpAddress
* test_jsonrpc_store: capture exception
* test_rln*: handling exceptions
* adaptation to make test_rln_* to work properly
* signature enhancement of group_manager methods
2023-12-14 07:16:39 +01:00
Ivan FB
0cc0c80563
chore: peer_manager.nim - reduce logs from debug to trace (#2279) 2023-12-12 16:00:18 +01:00
Aaryamann Challani
15c1f974df
fix(rln-relay): graceful retries on rpc calls (#2250)
* fix(rln-relay): graceful retries on rpc calls

* fix: missing file
2023-12-11 14:59:16 +05:30
Ivan FB
77c5ba7669
archive: simplify and enhance async retention policy application (#2278)
* Avoid using timer and just use an infinite async loop that can be
cancelled at any time.
2023-12-11 08:50:40 +01:00
Simon-Pierre Vivier
281c13a429
added sharded peer store pruning (#2167) 2023-12-07 07:21:18 -05:00
Simon-Pierre Vivier
dba9820c1f
feat: shard aware peer management (#2151) 2023-12-07 06:48:28 -05:00
Ivan FB
42f1957920
fix: add protection in rest service to always publish with timestamp if user doesn't provide it (#2261) 2023-12-06 14:02:21 +01:00
Sasha
103d3981ad
fix: remove trailing commas from keystore json (#2200)
* fix: remove trailing commas from keystore json

* keyfile.nim: try a different Json formatting approach

* build keystore

* address comment

---------

Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2023-12-01 12:57:19 +01:00
Ivan FB
89dbbd2048
waku_store/common.nim: correct ret code in PEER_DIAL_FAILURE (#2260) 2023-11-30 16:42:58 +01:00
Ivan FB
72a1f8c724
chore: Better feedback invalid content topic (#2254)
* typo correction appplication -> application
* content_topic.nim: better feedback to user when wrong topic is passed
* test_namespaced_topics.nim: updating tests accordingly
2023-11-30 11:11:33 +01:00
omahs
958b9bd712
chore: fix typos (#2239) 2023-11-30 11:08:08 +01:00
Ivan FB
6301bec05d
fix: waku_filter_v2/common: PEER_DIAL_FAILURE ret code change: 200 -> 504 (#2236) 2023-11-30 10:47:45 +01:00
Simon-Pierre Vivier
bebaa59c3b
chore(rest): refactor message cache (#2221) 2023-11-28 07:21:41 -05:00
Simon-Pierre Vivier
c301e880c8
chore: refactoring peer storage (#2243) 2023-11-27 08:08:58 -05:00
Ivan FB
b31c182325
chore: postres driver allow setting the max number of connection from a parameter (#2246)
* postres driver: allow setting the max number of connections from a parameter
2023-11-24 16:21:22 +01:00
Abhimanyu
a1ed517f9c
fix: extended Postgres code to support retention policy + refactoring (#2244)
* updated Postgres retention policy code + refactoring

* Update waku/waku_archive/driver/postgres_driver/postgres_driver.nim

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>

* updated code review changes

* data unit fixed, processing everything in bytes now

---------

Co-authored-by: Simon-Pierre Vivier <simvivier@status.im>
2023-11-24 15:43:47 +01:00
Abhimanyu
aeb77a3e75
feat: Add new DB column messageHash (#2202)
* feat: added DB column messageHash

* feat: minor change

* feat: minor merge conflict fix

* Update test_resume.nim

* Update test_resume.nim

* randomblob() func used to populate attribute

* PRIMARY key updated - SQLite and Postgres
2023-11-22 17:32:56 +01:00
Abhimanyu
a22ee60494
chore: deterministic message hash algorithm updated (#2233)
* deterministic hash algorithm updated + testcases

* updated code review
2023-11-22 15:23:43 +01:00
gabrielmer
59ee3c69c8
chore(REST): returning lightpush support and updated filter protocol (#2219) 2023-11-22 10:56:23 +02:00
Ivan FB
c48accb513
waku_store: better response when the store is requested with wrong cursor (#2231) 2023-11-22 09:32:39 +01:00
Simon-Pierre Vivier
897f487978
chore: mics. improvements to cluster id and shards setup (#2187) 2023-11-21 15:15:39 -05:00
Alvaro Revuelta
51f36099d5
fix(rln): error in api when rate limit (#2212) 2023-11-21 19:24:31 +01:00
Ivan FB
c973b85069
peer_manager.nim: better feedback if can't dial peer with WakuMetadataCodec (#2230) 2023-11-21 14:54:45 +01:00
Abhimanyu
9ff441ab47
chore: removing automatic vacuuming from retention policy code (#2228)
* retention policy and testcase updated

* removing dead code

* review updated code
2023-11-21 11:27:50 +01:00
Ivan FB
13fb7c2f74
group_manager.nim more except detail when cant connect eth client (#2195) 2023-11-20 23:25:55 +01:00
gabrielmer
ef8ffbdbd0
chore: decoupling announced and listen addresses (#2203) 2023-11-16 18:15:27 +02:00
Álex Cabeza Romero
f5f431382b
test(waku-relay): Relay (#2101)
* Implement message id tests.
* Implement relay tests.
* Update import paths to use test_all.
2023-11-15 16:11:36 +01:00
Álex Cabeza Romero
85265e6c61
test(waku-filter): Unsubscribe tests (#2085)
* Implement unsubscribe waku filter tests.
* test(waku-filter): Unsubscribe all, payloads and security tests (#2095)
* Implement waku node filter Security and Privacy tests (#2096)
2023-11-15 10:26:01 +01:00
NagyZoltanPeter
6d81e3841a
chore: Allow text/plain content type descriptor for json formatted content body (#2209)
* Allow text/plain content type descriptor for json formatted content body. Refactored duplicated encode/decode functions for rest api

* Fix relay endpoint decodings of content bodies to accept text/plain

* Added support for content body decoder for checking media type if additional parameters are present

* Fix wrong usage of ContentTypeData - appeared only for tests
2023-11-14 16:59:53 +01:00