f54ba10bc7
chore(archive): archive and drivers refactor ( #2761 )
...
* queue driver refactor (#2753 )
* chore(archive): archive refactor (#2752 )
* chore(archive): sqlite driver refactor (#2754 )
* chore(archive): postgres driver refactor (#2755 )
* chore(archive): renaming & copies (#2751 )
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
The id column is needed because it contains the message digest
which is used in store v2, and we need to keep support to
store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
We want the nwaku node to simultaneously support store-v2
requests and store-v3 requests.
Only the legacy archive is in charge of archiving messages, and the
archived information is suitable to fulfill both store-v2 and
store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2024-07-12 18:19:12 +02:00
1c9eb27415
fix(rln_keystore_generator): improve error handling for unrecoverable failure ( #2881 )
2024-07-10 19:12:49 +02:00
4ac4ab2a41
chore: sqlite make sure code is always run ( #2891 )
2024-07-10 18:34:28 +02:00
e1518cf9ff
chore: deprecating named sharding ( #2723 )
2024-07-09 18:36:12 +03:00
d0980eba4c
setting connectivity loop interval to 30 seconds ( #2878 )
2024-07-09 17:33:18 +03:00
9576b77c76
postgres_driver: better partition creation without exclusive access ( #2887 )
2024-07-09 13:41:29 +02:00
fd6a71cdd7
chore: Bump dependencies for v0.31.0 ( #2885 )
...
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
2024-07-09 13:14:28 +02:00
8bfad3ab45
chore: refactor relative path to better absolute ( #2861 )
2024-07-06 00:03:38 +02:00
f7c4a9e09e
Merge branch 'master' into update-master-from-release-v30
2024-07-03 22:52:15 +02:00
3fc0994bd0
chore: use sepolia testnet ( #2872 )
2024-07-03 18:11:11 +02:00
519abbebf0
Merge branch 'release/v0.30' into update-master-from-release-v0.30
2024-07-02 10:16:40 +02:00
cae0c7e378
chore: saving agent and protoVersion in peerStore ( #2860 )
2024-07-01 13:29:14 +02:00
31c632e427
chore: unit test for duplicate message push ( #2852 )
...
* chore: add unit test for testing duplicate message push with timedcache
* chore: update according to better naming convention
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2024-06-28 18:16:06 +05:30
9bd8c33aee
chore: remove all pre-nim-1.6 deadcode from codebase ( #2857 )
2024-06-28 16:04:57 +05:30
05356ab97f
chore(rln-relay): add chain-id flag to wakunode and restrict usage if mismatches rpc provider ( #2858 )
2024-06-28 11:19:16 +02:00
dad054d6b3
fix(rln): nullifierlog vulnerability ( #2855 )
2024-06-28 09:25:10 +02:00
8f14c04735
feat: Added proper per shard bandwidth metric calculation ( #2851 )
...
* Added proper per shard bandwidth metric calculation and proper logging of in/out messages
Changed rate limit metrics for dashboard
Updated monitoring dashboard for bw and rate metrics
2024-06-28 02:48:29 +02:00
76d5b2642d
chore: ignore arbitrary data stored in `multiaddrs` enr key ( #2853 )
2024-06-27 10:01:47 -04:00
7b97c8003c
chore: add TWN parameters for RLNv2 ( #2843 )
2024-06-27 11:45:21 +02:00
7205f95cff
chore: adding origin to peers admin endpoint ( #2848 )
2024-06-26 17:59:12 +02:00
974b8a39ac
chore: adding discv5 logs ( #2811 )
2024-06-26 14:25:58 +02:00
a05fa06919
chore: archive.nim - increase the max limit of content topics per query to 100 ( #2846 )
2024-06-26 12:24:15 +02:00
99149ea9dc
fix: duplicate message forwarding in filter service ( #2842 )
...
* fix: it's resolve duplicate message forwarding for filter service
* chore: update little flow
* fix: update implementation using timed cache method
* chore: simple format change
* chore: simple format change
* chore: update put function location
* chore: update according suggestion
2024-06-26 01:05:03 +05:30
c56785327a
fix(rln-relay): clear nullifier log only if length is over max epoch gap ( #2836 )
2024-06-24 13:29:56 +02:00
01050138c6
fix: only set disconnect time on left event ( #2831 )
2024-06-24 10:20:09 +02:00
733edae435
chore: update content-topic parsing for filter ( #2835 )
...
* chore: update content parsing for filter
* chore: update according to suggestion
* chore: update according to suggestion
2024-06-21 17:47:44 +05:30
94947a8504
chore: better descriptive log ( #2826 )
...
* chore: update logs with topic discription & debug msg
* chore: update unit according to error msg
* chore: update rest unit according to error msg
* chore: add content-topic with debug msg
2024-06-20 18:38:55 +05:30
a02832fe12
feat(rlnv2): clean fork of rlnv2 ( #2828 )
...
* chore(rlnv2): contract interface changes (#2770 )
* fix: tests
* fix: remove stuint[32]
* chore(submodule): update zerokit submodule to v0.5.1 (#2782 )
* fix: remove cond comp for lightpush test
* fix: ci and nonceManager
2024-06-20 14:55:50 +02:00
41bc582823
stop connecting to out peers until target is reached ( #2823 )
2024-06-20 12:16:15 +02:00
dd3c13582e
feat(rlnv2): clean fork of rlnv2 ( #2828 )
...
* chore(rlnv2): contract interface changes (#2770 )
* fix: tests
* fix: remove stuint[32]
* chore(submodule): update zerokit submodule to v0.5.1 (#2782 )
* fix: remove cond comp for lightpush test
* fix: ci and nonceManager
2024-06-20 11:35:21 +02:00
325e131697
fix: adding peer exchange peers to the peerStore ( #2824 )
2024-06-20 10:46:40 +02:00
cda18f96ca
fix: update peers ENRs in peer store in case they are updated ( #2818 )
2024-06-19 17:29:55 +02:00
a7d2dfd0aa
fix: revert "chore: adding observers for message logging ( #2800 )" ( #2815 )
2024-06-17 14:30:30 +02:00
93e9ba22aa
fix: revert "chore: adding observers for message logging ( #2800 )" ( #2815 )
2024-06-17 13:14:05 +02:00
3b27aee820
fix: mount metadata in wakucanary ( #2793 )
...
* chore: integrate cluster id and shards to waku node.
2024-06-14 18:29:42 +05:30
ab4b5b27bd
postgres_driver: better sync lock in partition creation ( #2809 )
...
With the original approach it happened cases where one connection
acquired the lock and other connection tried to release the same lock,
causing an unrecoverable failure which made the node to end.
2024-06-14 10:07:41 +02:00
0561e5bd45
feat: RLN proofs as a lightpush service ( #2768 )
2024-06-13 21:10:00 +04:00
b52286524b
chore: adding observers for message logging ( #2800 )
2024-06-13 18:35:56 +02:00
5989de88a4
fix: multi nat initialization causing dead lock in waku tests + serialize test runs to avoid timing and port occupied issues ( #2799 )
...
* Prevent multiple nat module initialization that cause dead lock in nat refresh thread tear down during tests.
* NPROC to 1 to avoid parallel test runs can lead to timing and port allocation issues
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
2024-06-12 07:49:55 +02:00
25a8b98f12
Remove filterTimeout configuration option as it remained after removing legacy filter where it belong to. ( #2797 )
2024-06-11 14:07:46 +02:00
f5d87c5b23
chore: set msg_hash logs to notice level ( #2737 )
2024-06-10 15:56:55 +02:00
55a87d21d6
fix: more detailed logs to differentiate shards with peers ( #2794 )
2024-06-10 13:40:18 +05:30
31bd6d71c0
chore: Minor enhancements ( #2789 )
...
* archive.nim: reduce the database report interval from 1 to 30 min
This counts the number of rows with "select count(1) from messages"
which is quite intense and we shouldn't run it every minute
* aside cleanup
2024-06-09 23:09:23 +02:00
c5d19c4491
chore: postgres_driver - acquire/release advisory lock when creating partitions ( #2784 )
2024-06-07 17:54:26 +02:00
77adfccd93
chore: simple link refactor ( #2781 )
2024-06-07 13:07:15 +05:30
dbe79250da
postgres_driver: simple reformat with nph ( #2785 )
2024-06-06 12:04:40 +02:00
3a93437300
add new index to optimize the order by storedAt ( #2778 )
2024-06-06 11:38:58 +02:00
24661ea969
postgres partitions: ensure the partition ranges are o'clock ( #2776 )
...
Also, skip the error "partition already exists" because that happens
when multiple nodes interact with the same database.
2024-06-05 17:45:38 +02:00
21a4b5bd4a
unifying clusterId to be uint16 ( #2777 )
2024-06-05 15:32:35 +02:00
dfc979a8ef
chore: extract common prefixes into a constant for multiple query ( #2747 )
...
* chore: extract select to constant for multiple prefix
* chore: Add a space at the end of selectClause to facilitate better string merging.
Signed-off-by: DarshanBPatel <darshan@status.im>
---------
Signed-off-by: DarshanBPatel <darshan@status.im>
2024-06-03 21:52:53 +05:30
88b8e1867a
fix(waku_archive): only allow a single instance to execute migrations ( #2736 )
2024-05-31 12:08:16 -04:00
8b42f199ba
postgres_driver.nim: add missing meta field to select queries ( #2741 )
2024-05-29 22:13:16 +02:00
a65b13fcb8
fix: invalid cursor returning messages ( #2724 )
...
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-27 10:54:10 -04:00
40296f9dbc
fix: do not print the db url on error ( #2725 )
2024-05-23 18:37:04 -04:00
4a6ec468db
fix: store v3 bug fixes ( #2718 )
2024-05-23 08:01:52 -04:00
1e2aa57a26
chore: android support ( #2554 )
2024-05-21 21:00:22 -04:00
7464684842
chore: Discovery in libwaku ( #2711 )
...
* cwaku_example: add discoveryv5-discovery bool option
* libwaku: implement discovery capabilities
* node_lifecycle_request.nim: better control of possible errors when parsing config
2024-05-21 18:37:50 +02:00
176c98c589
standardize store types by using camel case instead of snake case ( #2709 )
2024-05-17 16:56:54 +02:00
3faffdbcaa
chore: libwaku - allow to properly set the log level in libwaku and unify a little ( #2708 )
...
* waku.nimble: set properly chronicles compilation flags for static libwaku
* adapt examples to new log setup
2024-05-17 16:28:54 +02:00
401630ee24
chore: waku_discv5, peer_manager - add more logs help debug discovery issues ( #2705 )
2024-05-16 22:30:51 +02:00
78132dc12e
chore: generic change to reduce the number of compilation warnings ( #2696 )
2024-05-16 22:29:11 +02:00
9dfdfa2774
feat: Added message size check before relay for lightpush ( #2695 )
2024-05-15 14:13:13 +04:00
3a2caaec6f
refactor shard parsing ( #2699 )
2024-05-14 20:17:17 +02:00
840e012294
chore: move code from wakunode2 to a more generic place, waku ( #2670 )
...
* testlib/wakunode.nim: not use cluster-id == 1 to avoid test rln by default
2024-05-13 17:45:48 +02:00
49dd6c1989
test(peer-and-connection-management): Implement tests ( #2566 )
...
* Implement peer and connection management tests.
* Fix multiple peers added on initialisation.
* Remove clusterId parameter from newTestWakuNode.
2024-05-13 17:25:44 +02:00
7d4857ea6c
chore: closing ping streams ( #2692 )
2024-05-13 12:07:57 +02:00
8451cf8e1b
chore: Postgres enhance get oldest timestamp ( #2687 )
...
* postgres: consider also the existing paritions when getting oldest timestamp
* test_driver_postgres_query: adapt test to oldest timestamp
2024-05-10 18:31:01 +02:00
be5471c6f3
feat: adding json string support to bindings config ( #2685 )
2024-05-10 10:56:17 +02:00
026d804a0d
feat: Added flexible rate limit checks for store, legacy store and lightpush ( #2668 )
...
* Added flexible rate limit checks for store, legacy store and lightpush. Also added rate and traffic metrics.
* Fix chat2 after WakuLegacyStoreCodec rename
* Update waku/common/ratelimit.nim
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Update waku/common/ratelimit.nim
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Update waku/waku_store_legacy/protocol.nim
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Fix review findings, added limit to debug logs
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-09 20:07:49 +02:00
fa26d05f8e
fix: message cache removal crash ( #2682 )
2024-05-09 10:38:55 -04:00
d700006a6a
feat: store v3 return pubsub topics ( #2676 )
2024-05-08 15:35:56 -04:00
6a1af92276
chore(rln-relay): health check should account for window of roots ( #2664 )
...
* test(rln-relay): health check should account for window of roots
* fix: some type-fu
* fix: widen the type vs narrowing
* fix: add extra parens
2024-05-08 17:48:44 +05:30
b46226fca0
postgres_driver: delete partitions in time retention policy ( #2679 )
2024-05-07 23:42:01 +02:00
82f95999cd
fix: add `meta` to sqlite migration scripts ( #2675 )
2024-05-07 09:39:06 -04:00
22f64bbd44
chore: updating TWN bootstrap fleet to waku.sandbox ( #2638 )
2024-05-07 13:37:17 +02:00
a46d4451eb
feat: supporting meta field in store ( #2609 )
2024-05-06 10:20:21 +02:00
f8184a6de8
postgres_driver.nim: debug -> trace put PotsgresDriver ( #2667 )
2024-05-03 17:41:14 +02:00
cee020f26d
fix(filter): log is too large ( #2665 )
2024-05-03 19:05:24 +05:30
2463527b24
refactor: big refactor to add waku component in libwaku instead of onlu waku node ( #2658 )
2024-05-03 14:07:15 +02:00
853ec1869e
fix: issue #2644 properly ( #2663 )
2024-05-03 13:40:20 +05:30
404810aa8d
refactor: simplify app.nim and move discovery items to appropriate modules ( #2657 )
2024-05-01 21:13:08 +02:00
e03d1165e6
fix: store v3 validate cursor & remove messages ( #2636 )
2024-05-01 14:47:06 -04:00
5dd645cfc3
fix(waku_keystore): sigsegv on different appInfo ( #2654 )
...
* fix(waku_keystore): sigsegv on different appInfo
* fix: field specific errors
* fix: more verbose error logs
2024-05-01 23:05:22 +05:30
d5e0e4a9b1
chore: log enhancement for message reliability analysis ( #2640 )
...
* log enhancement for message reliability analysis
The next modules are touched:
- waku_node.nim
- archive.nim
- waku_filter_v2/protocol.nim
- waku_relay/protocol.nim
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
2024-05-01 10:25:33 +02:00
a9e19efd7a
fix(rln-relay): persist metadata every batch during initial sync ( #2649 )
...
* fix(rln-relay): persist metadata every batch during initial sync
* fix: test
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* patch: isOkOr template
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-04-30 18:52:47 +05:30
4a110f65ff
refactor: metrics server. Simplify app.nim module ( #2650 )
2024-04-30 15:07:17 +02:00
6382dedb42
waku_node: first of all stop the waku-switch when stopping the waku-node ( #2651 )
...
This is aimed to avoid having flaky tests
2024-04-30 12:52:11 +02:00
069b23f67e
waku_node.nim: simplify stop proc ( #2645 )
...
There is no need to explicitly stop mounted libp2p protocols
because they are already being stopped after the switch.stop()
is being called
2024-04-29 17:47:18 +02:00
28389cf043
rest/store/types: contentTopic -> content_topic ( #2646 )
2024-04-29 16:19:07 +02:00
8d1b0834db
fix: handle named sharding in enr ( #2647 )
2024-04-29 18:53:49 +05:30
6e6cb298a3
fix: parse shards properly in enr config for non twn ( #2633 )
2024-04-26 17:51:52 +05:30
2a4c0f1543
chore: Enabling to use a full node for lightpush via rest api without lightpush client configured ( #2626 )
...
* Enabling to use a full node for lightpush via rest api without light push client configured
2024-04-26 12:42:47 +02:00
1d7ff2881b
chore(rln-relay): resultify rln-relay 1/n ( #2607 )
...
* chore(rln-relay): resultify rln-relay 1/n
* fix: v2 too
* fix: for static group manager
* fix: cleanup, make PR digestable
* fix: remove resultified retry wrapper
* fix: cleanup
* fix: cleanup
2024-04-26 11:53:58 +02:00
843fe217fa
fix: proto field numbers & status desc ( #2632 )
2024-04-25 15:43:21 -04:00
5f65565c85
fix: missing rate limit setting for legacy store protocol ( #2631 )
2024-04-25 17:51:34 +02:00
0b0fbfad5c
feat: store v3 ( #2431 )
2024-04-25 09:09:52 -04:00
9c9883a630
fix(rln-relay): enforce error callback to remove exception raised from retryWrapper ( #2622 )
2024-04-24 17:11:22 +02:00
b28207ab07
refactor: addition of waku_api/rest/builder.nim and reduce app.nim ( #2623 )
2024-04-24 15:59:50 +02:00
6d135b0d1b
chore: Separation of node health and initialization state from rln_relay ( #2612 )
...
* Separation of node health and initialization state from rln_relay status. Make (only) health endpoint avail early and install others in the last stage of node setup.
* Proper json report from /health, adjusted and fixed test, added convenient script for checking node health
* Stop wakunode2 if configured rest server cannot be started
* Fix wakuRlnRelay protocol existence check
* Fix typo
* Removed unused imports from touched files.
* Added missing /health test for all
2024-04-23 18:53:18 +02:00
1a23700d54
fix(rln-relay): increase retries for 1 minute recovery time ( #2614 )
2024-04-23 15:11:14 +02:00
Simon-Pierre Vivier
Aaryamann Challani
Ivan FB
gabrielmer
Darshan K
Alvaro Revuelta
Ivan Folgueira Bande
NagyZoltanPeter
richΛrd
Akhil
Darshan K
Álex Cabeza Romero
Prem Chaitanya Prathi