53 Commits

Author SHA1 Message Date
Ivan FB
7611137c1f
fix: apply modulus in sharding (#3530) 2025-08-03 17:28:28 +02:00
fryorcraken
16bda047e1
chore!: make sharding configuration explicit (#3468)
* Reserve `networkconfig` name to waku network related settings

* Rename cluster conf to network conf

 A `NetworkConf` is a Waku network configuration.

# Conflicts:
#	tests/factory/test_waku_conf.nim

# Conflicts:
#	tests/factory/test_waku_conf.nim

* Improve sharding configuration

A smarter data types simplifies the logic.

* Fixing tests

* fixup! rename to endpointConf

* wip: autosharding is a specific configuration state and treat it like
it

# Conflicts:
#	waku/factory/external_config.nim

* refactor lightpush handler

some metrics error reporting were missing

# Conflicts:
#	waku/waku_lightpush/protocol.nim

* test_node_factory tests pass

* remove warnings

* fix tests

* Revert eager previous replace-all command

* fix up build tools compilation

* metadata is used to store cluster id

* Mount relay routes in static sharding

* Rename activeRelayShards to subscribeShards

To make it clearer that these are the shards the node will subscribe to.

* Remove unused msg var

* Improve error handling

* Set autosharding as default, with 1 shard in network

Also makes shards to subscribe to all shards in auto sharding, none in
static sharding.
2025-07-04 17:10:53 +10:00
NagyZoltanPeter
3db00f39ee
chore: extended /admin/v1 RESP API with different option to look at current connected/relay/mesh state of the node (#3382)
* Extended /admin/v1 RESP API with different option to look at current connected/relay/mesh state of the node
* Added score information for peer info retrievals
2025-04-24 08:36:02 +02:00
Simon-Pierre Vivier
bf735e7779
fix: waku sync 2.0 codecs ENR support (#3326) 2025-03-14 12:01:11 -04:00
NagyZoltanPeter
e0b563ffe5
feat: lightpush v3 (#3279)
* Separate new lightpush protocol
New RPC defined
Rename al occurence of old lightpush to legacy lightpush, fix rest tests of lightpush
New lightpush protocol added back
Setup new lightpush protocol, mounting and rest api for it

	modified:   apps/chat2/chat2.nim
	modified:   tests/node/test_wakunode_lightpush.nim
	modified:   tests/node/test_wakunode_sharding.nim
	modified:   tests/test_peer_manager.nim
	modified:   tests/test_wakunode_lightpush.nim
	renamed:    tests/waku_lightpush/lightpush_utils.nim -> tests/waku_lightpush_legacy/lightpush_utils.nim
	renamed:    tests/waku_lightpush/test_all.nim -> tests/waku_lightpush_legacy/test_all.nim
	renamed:    tests/waku_lightpush/test_client.nim -> tests/waku_lightpush_legacy/test_client.nim
	renamed:    tests/waku_lightpush/test_ratelimit.nim -> tests/waku_lightpush_legacy/test_ratelimit.nim
	modified:   tests/wakunode_rest/test_all.nim
	renamed:    tests/wakunode_rest/test_rest_lightpush.nim -> tests/wakunode_rest/test_rest_lightpush_legacy.nim
	modified:   waku/factory/node_factory.nim
	modified:   waku/node/waku_node.nim
	modified:   waku/waku_api/rest/admin/handlers.nim
	modified:   waku/waku_api/rest/builder.nim
	new file:   waku/waku_api/rest/legacy_lightpush/client.nim
	new file:   waku/waku_api/rest/legacy_lightpush/handlers.nim
	new file:   waku/waku_api/rest/legacy_lightpush/types.nim
	modified:   waku/waku_api/rest/lightpush/client.nim
	modified:   waku/waku_api/rest/lightpush/handlers.nim
	modified:   waku/waku_api/rest/lightpush/types.nim
	modified:   waku/waku_core/codecs.nim
	modified:   waku/waku_lightpush.nim
	modified:   waku/waku_lightpush/callbacks.nim
	modified:   waku/waku_lightpush/client.nim
	modified:   waku/waku_lightpush/common.nim
	modified:   waku/waku_lightpush/protocol.nim
	modified:   waku/waku_lightpush/rpc.nim
	modified:   waku/waku_lightpush/rpc_codec.nim
	modified:   waku/waku_lightpush/self_req_handler.nim
	new file:   waku/waku_lightpush_legacy.nim
	renamed:    waku/waku_lightpush/README.md -> waku/waku_lightpush_legacy/README.md
	new file:   waku/waku_lightpush_legacy/callbacks.nim
	new file:   waku/waku_lightpush_legacy/client.nim
	new file:   waku/waku_lightpush_legacy/common.nim
	new file:   waku/waku_lightpush_legacy/protocol.nim
	new file:   waku/waku_lightpush_legacy/protocol_metrics.nim
	new file:   waku/waku_lightpush_legacy/rpc.nim
	new file:   waku/waku_lightpush_legacy/rpc_codec.nim
	new file:   waku/waku_lightpush_legacy/self_req_handler.nim

Adapt to non-invasive libp2p observers

cherry pick latest lightpush (v1) changes into legacy lightpush code after rebase to latest master

Fix vendor dependencies from origin/master after failed rebase of them

Adjust examples, test to new lightpush - keep using of legacy

Fixup error code mappings

Fix REST admin interface with distinct legacy and new lightpush

Fix lightpush v2 tests

* Utilize new publishEx interface of pubsub libp2p

* Adapt to latest libp2p pubslih design changes. publish returns an outcome as Result error.

* Fix review findings

* Fix tests, re-added lost one

* Fix rebase

* Apply suggestions from code review

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Addressing review comments

* Fix incentivization tests

* Fix build failed on libwaku

* Change new lightpush endpoint version to 3 instead of 2. Noticed that old and new lightpush metrics can cause trouble in monitoring dashboards so decided to give new name as v3 for the new lightpush metrics and change legacy ones back - temporarly till old lightpush will be decommissioned

* Fixing flaky test with rate limit timing

* Fixing logscope of lightpush and legacy lightpush

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2025-03-05 12:07:56 +01:00
gabrielmer
9c209b4c3b
chore: sending msg hash as string on libwaku message event (#3234) 2025-01-30 10:15:31 +02:00
Simon-Pierre Vivier
29fda2dab7
feat: waku store sync 2.0 common types & codec (#3213) 2025-01-22 11:08:23 -05:00
NagyZoltanPeter
beb21c78f4
fix: lite-protocol-tester receiver exit check (#3187)
* Fix receiver exit criteria, not let it wait forever in some cases, added a timely check from the last arrived message
* Extend dial and service usage failure metrics with agent string to reveal service nodes origins
* Adjusted infra testing content topic to be unique in the system
* Extend error logs with peer's agent string, fix exit criteria
* Add informative log for not waiting for more messages
* Add unknown as default for empty agent identifier
* better explain exit logic of receiver
* Address review comment - checking for last message arrival return Optional Moment instead of result - better explains what is happening.
2024-12-07 01:22:50 +01:00
richΛrd
3665991a65
chore: support ping with multiple multiaddresses and close stream (#3154) 2024-10-28 15:51:07 -04:00
Ivan FB
cfde7eea82
chore: Circuit relay (#3112)
* undo apt install libpcre (not circuit-relay related.)
* nat.nim: protect against possible exceptions when calling getExternalIP
* new external CLI argument, isRelayClient
* waku factory change to mount circuit hop proto by default
* waku_node: move autonat_service to a separate module
2024-10-28 09:17:46 +01:00
gabrielmer
b4d3678ad9
fix: linting error (#3146) 2024-10-25 11:04:17 +03:00
gabrielmer
69d9524fa4
chore: saving peers enr capabilities (#3127) 2024-10-24 15:31:04 +03:00
Simon-Pierre Vivier
a386880be9
fix: add more logs, stagger intervals & set prune offset to 10% for waku sync (#3142) 2024-10-23 17:56:19 -04:00
Darshan K
729e63f53d
refactor: wrap peer store (#3051)
Encapsulate peerstore with wakupeerstore
2024-09-27 18:16:46 +05:30
gabrielmer
b358c90fa5
chore: disabling metrics for libwaku (#3058) 2024-09-25 14:08:01 +03:00
gabrielmer
a3cd2a1a92
chore: deprecating pubsub topic (#2997) 2024-09-10 15:07:12 -06:00
gabrielmer
67439057fb
chore: rename NsPubsubTopic (#2974) 2024-08-19 11:29:35 +02:00
richΛrd
2fd4eb62fd
chore: print WakuMessageHash as hex strings (#2969) 2024-08-14 21:04:20 +02:00
Simon-Pierre Vivier
2cc86c51da
feat: Nwaku Sync (#2403)
* feat: Waku Sync Protocol

* feat: state machine (#2656)

* feat: pruning storage mehcanism (#2673)

* feat: message transfer mechanism & tests (#2688)

* update docker files

* added ENR filed for sync & misc. fixes

* adding new sync range param & fixes

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com>
2024-08-13 07:27:34 -04:00
gabrielmer
e1518cf9ff
chore: deprecating named sharding (#2723) 2024-07-09 18:36:12 +03:00
Ivan FB
fd6a71cdd7
chore: Bump dependencies for v0.31.0 (#2885)
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
  Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
  not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
2024-07-09 13:14:28 +02:00
Darshan K
9bd8c33aee
chore: remove all pre-nim-1.6 deadcode from codebase (#2857) 2024-06-28 16:04:57 +05:30
Darshan K
94947a8504
chore: better descriptive log (#2826)
* chore: update logs with topic discription & debug msg

* chore: update unit according to error msg

* chore: update rest unit according to error msg

* chore: add content-topic with debug msg
2024-06-20 18:38:55 +05:30
gabrielmer
21a4b5bd4a
unifying clusterId to be uint16 (#2777) 2024-06-05 15:32:35 +02:00
Simon-Pierre Vivier
4a6ec468db
fix: store v3 bug fixes (#2718) 2024-05-23 08:01:52 -04:00
richΛrd
1e2aa57a26
chore: android support (#2554) 2024-05-21 21:00:22 -04:00
Ivan FB
78132dc12e
chore: generic change to reduce the number of compilation warnings (#2696) 2024-05-16 22:29:11 +02:00
Simon-Pierre Vivier
e03d1165e6
fix: store v3 validate cursor & remove messages (#2636) 2024-05-01 14:47:06 -04:00
Simon-Pierre Vivier
0b0fbfad5c
feat: store v3 (#2431) 2024-04-25 09:09:52 -04:00
Prem Chaitanya Prathi
e61e4ff90a
fix: don't use WakuMessageSize in req/resp protocols (#2601)
* fix: don't use WakuMessageSize in req/resp protocols
2024-04-20 09:10:52 +05:30
Ivan FB
3ba4378cbe
Generic re-style with nph 0.5.1 (#2396) 2024-03-16 00:08:47 +01:00
Álex Cabeza Romero
f436240d53
test(peer-exchange): Implement peer exchange tests (#2464)
* Implement peer exchange tests.
* Refactor, and remove duplicated tests.
* feat(wakunode): Resultify fetch peer exchange peers (#2486)
2024-03-14 17:48:09 +01:00
Alvaro Revuelta
5a225809cd
fix: enable autosharding in any cluster (#2505) 2024-03-13 10:58:13 +01:00
Simon-Pierre Vivier
505479b870
feat: archive update for store v3 (#2451) 2024-03-12 07:51:03 -04:00
Simon-Pierre Vivier
0a75122828
fix: time on 32 bits architecture (#2492)
authored-by: Emil Ivanichkov <emil.ivanichkov@gmail.com>
2024-03-01 07:58:45 -05:00
Simon-Pierre Vivier
edca1df1a8
feat: shard aware relay peer management (#2332)
note that this feature is behind a config flag. `--relay-shard-manager`
2024-01-30 07:28:21 -05:00
Álex Cabeza Romero
3bbe234962
test(autosharding): Functional Tests (#2318)
* Implement autosharding tests.
2024-01-04 16:26:27 +01:00
Ivan FB
ed09074cc3
chore: message.nim - set max message size to 150KiB according to spec (#2298)
* message.nim: set max message size to 150KiB according to spec

Using KiB instead of KB because that seems more aligned with
the actual default defined in nim-libp2p (1024 * 1024)

Spec details: https://rfc.vac.dev/spec/64/#message-size

* test_protocol.nim: align test to current WakuMessage limit
* test_waku_client.nim: adapt test to MaxWakuMessageSize change
* make maxMessageSize configurable for wakunode2
* wakunode2 app now accepts max-num-bytes-msg-size with KiB, KB, or B units
* testlib/wakunode.nim: set maxMessageSize: "1024 KiB"
* test_waku_client.nim: remove duplicate check in "Valid Payload Sizes"
* set DefaultMaxWakuMessageSizeStr as the only source of truth
* external_config.nim: rename max-num-bytes-msg-size -> max-msg-size
2024-01-03 13:11:50 +01:00
Ivan FB
32668f43f5
fix: Revert "feat: shard aware peer management (#2151)" (#2312)
This reverts commit dba9820c1fa00f414f18d57f7a3ff38b67d2bb1a.

We need to revert this commit because
the waku-simulator stopped working. i.e. the nodes couldn't establish
connections among them: 054ba9e33f

Also, the following js-waku test fails due to this commit:
"same cluster, different shard: nodes connect"

* waku_lightpush/protocol.nim: minor changes to make it compile after revert
2023-12-20 15:23:41 +01:00
Simon-Pierre Vivier
dba9820c1f
feat: shard aware peer management (#2151) 2023-12-07 06:48:28 -05:00
Ivan FB
72a1f8c724
chore: Better feedback invalid content topic (#2254)
* typo correction appplication -> application
* content_topic.nim: better feedback to user when wrong topic is passed
* test_namespaced_topics.nim: updating tests accordingly
2023-11-30 11:11:33 +01:00
Abhimanyu
aeb77a3e75
feat: Add new DB column messageHash (#2202)
* feat: added DB column messageHash

* feat: minor change

* feat: minor merge conflict fix

* Update test_resume.nim

* Update test_resume.nim

* randomblob() func used to populate attribute

* PRIMARY key updated - SQLite and Postgres
2023-11-22 17:32:56 +01:00
Abhimanyu
a22ee60494
chore: deterministic message hash algorithm updated (#2233)
* deterministic hash algorithm updated + testcases

* updated code review
2023-11-22 15:23:43 +01:00
Simon-Pierre Vivier
a47dc9e663
fix: cluster id & sharding terminology (#2104) 2023-10-05 08:37:05 -04:00
NagyZoltanPeter
7b5c36b1c6
feat: /admin rest api endpoint (#2094)
/admin rest api implementation and tests
* open api doc
* Add rest admin test to all tests
* Enrich /admin get peers interface, group protocols by peers in response
2023-10-05 14:00:09 +02:00
Simon-Pierre Vivier
1763b1efa1
feat: Autosharding API for RELAY subscriptions (#1983) 2023-09-26 07:33:52 -04:00
NagyZoltanPeter
dac072f843
feat: HTTP REST API: Filter support v2 (#1890)
Filter v2 rest api support implemented 
Filter rest api documentation updated with v1 and v2 interface support.
Separated legacy filter rest interface
Fix code and tests of v2 Filter rest api
Filter v2 message push test added
Applied autoshard to Filter V2
Redesigned FilterPushHandling, code style, catch up apps and tests with filter v2 interface changes
Rename of FilterV1SubscriptionsRequest to FilterLegacySubscribeRequest, fix broken chat2 app, fix tests
Changed Filter v2 push handler subscription to simple register
Separate node's filterUnsubscribe and filterUnsubscribeAll
2023-09-14 21:28:57 +02:00
NagyZoltanPeter
563b2b20a5
chore: move SubscriptionManager under waku_core (#2025)
* chore: cherry-pick from Filter V2 RestApi PR: move FilterPushHandler and SubscriptionManager from Filter V1 to under waku_core
2023-09-12 17:08:40 +02:00
Aaryamann Challani
645b034367
feat(rln-relay): removed rln from experimental 🚀 (#2001)
* feat(rln-relay): removed rln from experimental 🚀

* fix(waku_node): if rln-relay is nil, return true

* chore(tests): mark rest health as skipped
2023-09-11 12:02:31 +05:30
Simon-Pierre Vivier
037b166263
fix: --topic should be ignore when using --pubsub-topic or --content-topic (#1977) 2023-08-31 16:13:45 -04:00