31 Commits

Author SHA1 Message Date
fryorcraken
16bda047e1
chore!: make sharding configuration explicit (#3468)
* Reserve `networkconfig` name to waku network related settings

* Rename cluster conf to network conf

 A `NetworkConf` is a Waku network configuration.

# Conflicts:
#	tests/factory/test_waku_conf.nim

# Conflicts:
#	tests/factory/test_waku_conf.nim

* Improve sharding configuration

A smarter data types simplifies the logic.

* Fixing tests

* fixup! rename to endpointConf

* wip: autosharding is a specific configuration state and treat it like
it

# Conflicts:
#	waku/factory/external_config.nim

* refactor lightpush handler

some metrics error reporting were missing

# Conflicts:
#	waku/waku_lightpush/protocol.nim

* test_node_factory tests pass

* remove warnings

* fix tests

* Revert eager previous replace-all command

* fix up build tools compilation

* metadata is used to store cluster id

* Mount relay routes in static sharding

* Rename activeRelayShards to subscribeShards

To make it clearer that these are the shards the node will subscribe to.

* Remove unused msg var

* Improve error handling

* Set autosharding as default, with 1 shard in network

Also makes shards to subscribe to all shards in auto sharding, none in
static sharding.
2025-07-04 17:10:53 +10:00
NagyZoltanPeter
9fe8ef8d27
feat: dynamic logging via REST API (#3451)
* Added /admin/v1/log-level/{logLevel} endpoint that is used for dynamic log level setting

credits to @darshankabariya co-authoring:
* Adapted conditional compile switch check from Darshan's solution

* formatting fix
2025-06-12 10:50:08 +02:00
NagyZoltanPeter
f4ad7a332e
chore: Added extra debug helper via getting peer statistics (#3443)
* Added extra debug helper via getting peer statistics on new /admin/v1/peers/stats endpoint

* Add /admin/v1/peers/stats client part

* Address review, change protocol names to codec string

* fix formatting
2025-06-10 02:10:06 +02:00
NagyZoltanPeter
f6fdd960f2
chore: return all peers from rest admin (#3395)
* Updated version of getting peers by /admin endpoints
2025-04-25 15:36:41 +02:00
NagyZoltanPeter
3db00f39ee
chore: extended /admin/v1 RESP API with different option to look at current connected/relay/mesh state of the node (#3382)
* Extended /admin/v1 RESP API with different option to look at current connected/relay/mesh state of the node
* Added score information for peer info retrievals
2025-04-24 08:36:02 +02:00
Miran
6ebb49a14b
chore: fix unused and deprecated imports (#3368) 2025-04-11 18:20:23 +03:00
Ivan FB
df58643ea3
chore: retrieve protocols in new added peer from discv5 (#3354)
* add new unit test to validate that any peer can be retrieved
* add new discv5 test and better peer store management
* wakuPeerStore -> switch.peerStore
* simplify waku_peer_store, better logs and peer_manager enhancements
2025-04-07 12:24:03 +02:00
NagyZoltanPeter
e0b563ffe5
feat: lightpush v3 (#3279)
* Separate new lightpush protocol
New RPC defined
Rename al occurence of old lightpush to legacy lightpush, fix rest tests of lightpush
New lightpush protocol added back
Setup new lightpush protocol, mounting and rest api for it

	modified:   apps/chat2/chat2.nim
	modified:   tests/node/test_wakunode_lightpush.nim
	modified:   tests/node/test_wakunode_sharding.nim
	modified:   tests/test_peer_manager.nim
	modified:   tests/test_wakunode_lightpush.nim
	renamed:    tests/waku_lightpush/lightpush_utils.nim -> tests/waku_lightpush_legacy/lightpush_utils.nim
	renamed:    tests/waku_lightpush/test_all.nim -> tests/waku_lightpush_legacy/test_all.nim
	renamed:    tests/waku_lightpush/test_client.nim -> tests/waku_lightpush_legacy/test_client.nim
	renamed:    tests/waku_lightpush/test_ratelimit.nim -> tests/waku_lightpush_legacy/test_ratelimit.nim
	modified:   tests/wakunode_rest/test_all.nim
	renamed:    tests/wakunode_rest/test_rest_lightpush.nim -> tests/wakunode_rest/test_rest_lightpush_legacy.nim
	modified:   waku/factory/node_factory.nim
	modified:   waku/node/waku_node.nim
	modified:   waku/waku_api/rest/admin/handlers.nim
	modified:   waku/waku_api/rest/builder.nim
	new file:   waku/waku_api/rest/legacy_lightpush/client.nim
	new file:   waku/waku_api/rest/legacy_lightpush/handlers.nim
	new file:   waku/waku_api/rest/legacy_lightpush/types.nim
	modified:   waku/waku_api/rest/lightpush/client.nim
	modified:   waku/waku_api/rest/lightpush/handlers.nim
	modified:   waku/waku_api/rest/lightpush/types.nim
	modified:   waku/waku_core/codecs.nim
	modified:   waku/waku_lightpush.nim
	modified:   waku/waku_lightpush/callbacks.nim
	modified:   waku/waku_lightpush/client.nim
	modified:   waku/waku_lightpush/common.nim
	modified:   waku/waku_lightpush/protocol.nim
	modified:   waku/waku_lightpush/rpc.nim
	modified:   waku/waku_lightpush/rpc_codec.nim
	modified:   waku/waku_lightpush/self_req_handler.nim
	new file:   waku/waku_lightpush_legacy.nim
	renamed:    waku/waku_lightpush/README.md -> waku/waku_lightpush_legacy/README.md
	new file:   waku/waku_lightpush_legacy/callbacks.nim
	new file:   waku/waku_lightpush_legacy/client.nim
	new file:   waku/waku_lightpush_legacy/common.nim
	new file:   waku/waku_lightpush_legacy/protocol.nim
	new file:   waku/waku_lightpush_legacy/protocol_metrics.nim
	new file:   waku/waku_lightpush_legacy/rpc.nim
	new file:   waku/waku_lightpush_legacy/rpc_codec.nim
	new file:   waku/waku_lightpush_legacy/self_req_handler.nim

Adapt to non-invasive libp2p observers

cherry pick latest lightpush (v1) changes into legacy lightpush code after rebase to latest master

Fix vendor dependencies from origin/master after failed rebase of them

Adjust examples, test to new lightpush - keep using of legacy

Fixup error code mappings

Fix REST admin interface with distinct legacy and new lightpush

Fix lightpush v2 tests

* Utilize new publishEx interface of pubsub libp2p

* Adapt to latest libp2p pubslih design changes. publish returns an outcome as Result error.

* Fix review findings

* Fix tests, re-added lost one

* Fix rebase

* Apply suggestions from code review

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Addressing review comments

* Fix incentivization tests

* Fix build failed on libwaku

* Change new lightpush endpoint version to 3 instead of 2. Noticed that old and new lightpush metrics can cause trouble in monitoring dashboards so decided to give new name as v3 for the new lightpush metrics and change legacy ones back - temporarly till old lightpush will be decommissioned

* Fixing flaky test with rate limit timing

* Fixing logscope of lightpush and legacy lightpush

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2025-03-05 12:07:56 +01:00
NagyZoltanPeter
e2b7149f82
chore: Bump nimbus and nim to latest available - nim-2.0.12 (#3188)
* Bump nimbus and nim to latest available - nim-2.0.12
* Fix name collision of templates of result.nim and nwaku serdes.nim - unrecognizedFieldWarning
2024-12-10 14:42:54 +01:00
Simon-Pierre Vivier
2ab9c3d363
feat: remove Waku Sync 1.0 & Negentropy (#3185) 2024-11-29 09:09:41 -05:00
gabrielmer
b3656d6eea
chore: naming connectPeer procedure (#3157) 2024-10-29 18:37:07 +02:00
Darshan K
729e63f53d
refactor: wrap peer store (#3051)
Encapsulate peerstore with wakupeerstore
2024-09-27 18:16:46 +05:30
Ivan FB
6c890de4a8
chore: Bump dependencies for v0.33 (#3017) 2024-09-09 10:45:14 +02:00
Simon-Pierre Vivier
2cc86c51da
feat: Nwaku Sync (#2403)
* feat: Waku Sync Protocol

* feat: state machine (#2656)

* feat: pruning storage mehcanism (#2673)

* feat: message transfer mechanism & tests (#2688)

* update docker files

* added ENR filed for sync & misc. fixes

* adding new sync range param & fixes

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com>
2024-08-13 07:27:34 -04:00
Sasha
a29eca77a0
chore: return all connected peers from REST API (#2923)
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2024-07-23 12:58:56 +02:00
Darshan K
9bd8c33aee
chore: remove all pre-nim-1.6 deadcode from codebase (#2857) 2024-06-28 16:04:57 +05:30
gabrielmer
7205f95cff
chore: adding origin to peers admin endpoint (#2848) 2024-06-26 17:59:12 +02:00
Ivan FB
78132dc12e
chore: generic change to reduce the number of compilation warnings (#2696) 2024-05-16 22:29:11 +02:00
NagyZoltanPeter
026d804a0d
feat: Added flexible rate limit checks for store, legacy store and lightpush (#2668)
* Added flexible rate limit checks for store, legacy store and lightpush. Also added rate and traffic metrics.

* Fix chat2 after WakuLegacyStoreCodec rename

* Update waku/common/ratelimit.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Update waku/common/ratelimit.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Update waku/waku_store_legacy/protocol.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Fix review findings, added limit to debug logs

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-09 20:07:49 +02:00
Simon-Pierre Vivier
0b0fbfad5c
feat: store v3 (#2431) 2024-04-25 09:09:52 -04:00
Alvaro Revuelta
e861317209
chore: remove deprecated legacy filter protocol (#2507)
* chore: remove deprecated legacy filter protocol

* fix: do not use legacy import in test

* fix: remove legacy test references

* fix: more test fixes, starting filter client

* fix: sigh. more references to remove.

* fix: fix dereferencing error

* fix: fix merge mess up

* fix: sigh. merge tool used tabs.

* fix: more peer manager tests needed fixing

---------

Co-authored-by: Hanno Cornelius <hanno@status.im>
Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2024-03-25 18:07:56 +00:00
Ivan FB
3ba4378cbe
Generic re-style with nph 0.5.1 (#2396) 2024-03-16 00:08:47 +01:00
Ivan FB
b890f9c663
rest: rm openapi defs. they are in https://github.com/waku-org/waku-rest-api (#2520) 2024-03-08 19:43:55 +01:00
Simon-Pierre Vivier
edca1df1a8
feat: shard aware relay peer management (#2332)
note that this feature is behind a config flag. `--relay-shard-manager`
2024-01-30 07:28:21 -05:00
NagyZoltanPeter
c3358409bb
chore: Refactor of FilterV2 subscription management with Time-to-live maintenance (#2341)
* Refactor of FilterV2 subscription handling and maintenance with addition subscription time-to-live support.
Fixed all tests and reworked where subscription handling changes needed it.
Adapted REST API /admin filter subscription retrieve to new filter subscription structure.

* Fix tests and PR comments

* Added filter v2 subscription timeout tests and fixed

* Fix review comments and suggestions. No functional change.

* Remove leftover echoes from test_rest_admin

* Fix failed legacy filter tests due to separation of mounting the filters.

* Small fixes, fix naming typo, removed duplicated checks in test
2024-01-16 17:27:40 +01:00
gabrielmer
bcab9d55da
feat: adding filter data admin endpoint (REST) (#2314) 2024-01-09 11:42:29 +01:00
Ivan FB
32668f43f5
fix: Revert "feat: shard aware peer management (#2151)" (#2312)
This reverts commit dba9820c1fa00f414f18d57f7a3ff38b67d2bb1a.

We need to revert this commit because
the waku-simulator stopped working. i.e. the nodes couldn't establish
connections among them: 054ba9e33f

Also, the following js-waku test fails due to this commit:
"same cluster, different shard: nodes connect"

* waku_lightpush/protocol.nim: minor changes to make it compile after revert
2023-12-20 15:23:41 +01:00
Simon-Pierre Vivier
dba9820c1f
feat: shard aware peer management (#2151) 2023-12-07 06:48:28 -05:00
gabrielmer
59ee3c69c8
chore(REST): returning lightpush support and updated filter protocol (#2219) 2023-11-22 10:56:23 +02:00
NagyZoltanPeter
6d81e3841a
chore: Allow text/plain content type descriptor for json formatted content body (#2209)
* Allow text/plain content type descriptor for json formatted content body. Refactored duplicated encode/decode functions for rest api

* Fix relay endpoint decodings of content bodies to accept text/plain

* Added support for content body decoder for checking media type if additional parameters are present

* Fix wrong usage of ContentTypeData - appeared only for tests
2023-11-14 16:59:53 +01:00
NagyZoltanPeter
7b5c36b1c6
feat: /admin rest api endpoint (#2094)
/admin rest api implementation and tests
* open api doc
* Add rest admin test to all tests
* Enrich /admin get peers interface, group protocols by peers in response
2023-10-05 14:00:09 +02:00