From 76feea39659658fd46382f975c545a7fd5950da3 Mon Sep 17 00:00:00 2001 From: Ivan Folgueira Bande Date: Tue, 11 Jun 2024 22:14:34 +0200 Subject: [PATCH 1/6] bump nim-libp2p from v1.2.0 to v1.3.0 --- vendor/nim-libp2p | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vendor/nim-libp2p b/vendor/nim-libp2p index c4da9be32..d0af3fbe8 160000 --- a/vendor/nim-libp2p +++ b/vendor/nim-libp2p @@ -1 +1 @@ -Subproject commit c4da9be32cc01efa2de066c396fe9ef1c7769aa1 +Subproject commit d0af3fbe8559f69195657a360c3dd4ac4552c811 From b775ae976fc5147a2e3d55aaa236caeaba98c3f1 Mon Sep 17 00:00:00 2001 From: Ivan Folgueira Bande Date: Tue, 11 Jun 2024 23:04:42 +0200 Subject: [PATCH 2/6] Update changelog for v0.29.0 --- CHANGELOG.md | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index addd3e0da..a59a660b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,62 @@ +## v0.29.0 (2024-06-11) + +## What's Changed + + +Notes: + +* Start deprecate named sharding ([PR](https://github.com/waku-org/nwaku/pull/2723)) + + +Release highlights: + +* Android support in libwaku +* Discovery is available in libwaku +* New LiteProcotolTester tool + +### Features + +- Push newly released nwaku image with latest-release tag ([#2732](https://github.com/waku-org/nwaku/issues/2732)) ([736ce1cb](https://github.com/waku-org/nwaku/commit/736ce1cb)) +- Rln-relay: use arkzkey variant of zerokit ([#2681](https://github.com/waku-org/nwaku/issues/2681)) ([e7b0777d](https://github.com/waku-org/nwaku/commit/e7b0777d)) + +### Bug Fixes + +- Increase on chain group manager starting balance ([#2795](https://github.com/waku-org/nwaku/issues/2795)) ([e72bb7e7](https://github.com/waku-org/nwaku/commit/e72bb7e7)) +- More detailed logs to differentiate shards with peers ([#2794](https://github.com/waku-org/nwaku/issues/2794)) ([55a87d21](https://github.com/waku-org/nwaku/commit/55a87d21)) +- waku_archive: only allow a single instance to execute migrations ([#2736](https://github.com/waku-org/nwaku/issues/2736)) ([88b8e186](https://github.com/waku-org/nwaku/commit/88b8e186)) +- Move postgres related tests under linux conditional ([57ecb3e0](https://github.com/waku-org/nwaku/commit/57ecb3e0)) +- Invalid cursor returning messages ([#2724](https://github.com/waku-org/nwaku/issues/2724)) ([a65b13fc](https://github.com/waku-org/nwaku/commit/a65b13fc)) +- Do not print the db url on error ([#2725](https://github.com/waku-org/nwaku/issues/2725)) ([40296f9d](https://github.com/waku-org/nwaku/commit/40296f9d)) +- Use `when` instead of `if` for adding soname on linux ([#2721](https://github.com/waku-org/nwaku/issues/2721)) ([cbaefeb3](https://github.com/waku-org/nwaku/commit/cbaefeb3)) +- Store v3 bug fixes ([#2718](https://github.com/waku-org/nwaku/issues/2718)) ([4a6ec468](https://github.com/waku-org/nwaku/commit/4a6ec468)) + +### Changes + +- Set msg_hash logs to notice level ([#2737](https://github.com/waku-org/nwaku/issues/2737)) ([f5d87c5b](https://github.com/waku-org/nwaku/commit/f5d87c5b)) +- Minor enhancements ([#2789](https://github.com/waku-org/nwaku/issues/2789)) ([31bd6d71](https://github.com/waku-org/nwaku/commit/31bd6d71)) +- postgres_driver - acquire/release advisory lock when creating partitions ([#2784](https://github.com/waku-org/nwaku/issues/2784)) ([c5d19c44](https://github.com/waku-org/nwaku/commit/c5d19c44)) +- Setting fail-fast to false in matrixed github actions ([#2787](https://github.com/waku-org/nwaku/issues/2787)) ([005349cc](https://github.com/waku-org/nwaku/commit/005349cc)) +- Simple link refactor ([#2781](https://github.com/waku-org/nwaku/issues/2781)) ([77adfccd](https://github.com/waku-org/nwaku/commit/77adfccd)) +- Improving liteprotocolteseter stats ([#2750](https://github.com/waku-org/nwaku/issues/2750)) ([4c7c8a15](https://github.com/waku-org/nwaku/commit/4c7c8a15)) +- Extract common prefixes into a constant for multiple query ([#2747](https://github.com/waku-org/nwaku/issues/2747)) ([dfc979a8](https://github.com/waku-org/nwaku/commit/dfc979a8)) +- wakucanary: fix fitler protocol, add storev3 ([#2735](https://github.com/waku-org/nwaku/issues/2735)) ([e0079cd0](https://github.com/waku-org/nwaku/commit/e0079cd0)) +- Bump nim-libp2p version ([#2661](https://github.com/waku-org/nwaku/issues/2661)) ([6fbab633](https://github.com/waku-org/nwaku/commit/6fbab633)) +- Link validation process docs to the release process file ([#2714](https://github.com/waku-org/nwaku/issues/2714)) ([ebe69be8](https://github.com/waku-org/nwaku/commit/ebe69be8)) +- Android support ([#2554](https://github.com/waku-org/nwaku/issues/2554)) ([1e2aa57a](https://github.com/waku-org/nwaku/commit/1e2aa57a)) +- Discovery in libwaku ([#2711](https://github.com/waku-org/nwaku/issues/2711)) ([74646848](https://github.com/waku-org/nwaku/commit/74646848)) +- libwaku - allow to properly set the log level in libwaku and unify a little ([#2708](https://github.com/waku-org/nwaku/issues/2708)) ([3faffdbc](https://github.com/waku-org/nwaku/commit/3faffdbc)) +- waku_discv5, peer_manager - add more logs help debug discovery issues ([#2705](https://github.com/waku-org/nwaku/issues/2705)) ([401630ee](https://github.com/waku-org/nwaku/commit/401630ee)) +- Generic change to reduce the number of compilation warnings ([#2696](https://github.com/waku-org/nwaku/issues/2696)) ([78132dc1](https://github.com/waku-org/nwaku/commit/78132dc1)) + +This release supports the following [libp2p protocols](https://docs.libp2p.io/concepts/protocols/): +| Protocol | Spec status | Protocol id | +| ---: | :---: | :--- | +| [`11/WAKU2-RELAY`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/11/relay.md) | `stable` | `/vac/waku/relay/2.0.0` | +| [`12/WAKU2-FILTER`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/12/filter.md) | `draft` | `/vac/waku/filter/2.0.0-beta1`
`/vac/waku/filter-subscribe/2.0.0-beta1`
`/vac/waku/filter-push/2.0.0-beta1` | +| [`13/WAKU2-STORE`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/13/store.md) | `draft` | `/vac/waku/store/2.0.0-beta4` | +| [`19/WAKU2-LIGHTPUSH`](https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/19/lightpush.md) | `draft` | `/vac/waku/lightpush/2.0.0-beta1` | +| [`66/WAKU2-METADATA`](https://github.com/waku-org/specs/blob/master/standards/core/metadata.md) | `raw` | `/vac/waku/metadata/1.0.0` | + ## v0.28.1 (2024-05-29) This patch release fixes the following bug: From 68c2e99a373fc5798da25e3c62a7d42b0e1924c2 Mon Sep 17 00:00:00 2001 From: Ivan Folgueira Bande Date: Fri, 14 Jun 2024 11:17:13 +0200 Subject: [PATCH 3/6] Update CHANGELOG.md for v0.29 --- CHANGELOG.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a59a660b0..f4d1ddcc3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,25 +2,27 @@ ## What's Changed - Notes: * Start deprecate named sharding ([PR](https://github.com/waku-org/nwaku/pull/2723)) - Release highlights: * Android support in libwaku * Discovery is available in libwaku * New LiteProcotolTester tool +* RLN proofs as a lightpush service ### Features +- RLN proofs as a lightpush service ([#2768](https://github.com/waku-org/nwaku/issues/2768)) ([0561e5bd](https://github.com/waku-org/nwaku/commit/0561e5bd)) - Push newly released nwaku image with latest-release tag ([#2732](https://github.com/waku-org/nwaku/issues/2732)) ([736ce1cb](https://github.com/waku-org/nwaku/commit/736ce1cb)) - Rln-relay: use arkzkey variant of zerokit ([#2681](https://github.com/waku-org/nwaku/issues/2681)) ([e7b0777d](https://github.com/waku-org/nwaku/commit/e7b0777d)) ### Bug Fixes +- Better sync lock in partition creation ([#2783](https://github.com/waku-org/nwaku/issues/2783)) ([8d3bbb1b](https://github.com/waku-org/nwaku/pull/2809/commits/8d3bbb1b4e79b15c8cf18bb91d366e9ec1153301)) +- Multi nat initialization causing dead lock in waku tests + serialize test runs to avoid timing and port occupied issues ([#2799](https://github.com/waku-org/nwaku/issues/2799)) ([5989de88](https://github.com/waku-org/nwaku/commit/5989de88)) - Increase on chain group manager starting balance ([#2795](https://github.com/waku-org/nwaku/issues/2795)) ([e72bb7e7](https://github.com/waku-org/nwaku/commit/e72bb7e7)) - More detailed logs to differentiate shards with peers ([#2794](https://github.com/waku-org/nwaku/issues/2794)) ([55a87d21](https://github.com/waku-org/nwaku/commit/55a87d21)) - waku_archive: only allow a single instance to execute migrations ([#2736](https://github.com/waku-org/nwaku/issues/2736)) ([88b8e186](https://github.com/waku-org/nwaku/commit/88b8e186)) @@ -30,8 +32,10 @@ Release highlights: - Use `when` instead of `if` for adding soname on linux ([#2721](https://github.com/waku-org/nwaku/issues/2721)) ([cbaefeb3](https://github.com/waku-org/nwaku/commit/cbaefeb3)) - Store v3 bug fixes ([#2718](https://github.com/waku-org/nwaku/issues/2718)) ([4a6ec468](https://github.com/waku-org/nwaku/commit/4a6ec468)) + ### Changes +- Adding observers for message logging ([#2800](https://github.com/waku-org/nwaku/issues/2800)) ([b5228652](https://github.com/waku-org/nwaku/commit/b5228652)) - Set msg_hash logs to notice level ([#2737](https://github.com/waku-org/nwaku/issues/2737)) ([f5d87c5b](https://github.com/waku-org/nwaku/commit/f5d87c5b)) - Minor enhancements ([#2789](https://github.com/waku-org/nwaku/issues/2789)) ([31bd6d71](https://github.com/waku-org/nwaku/commit/31bd6d71)) - postgres_driver - acquire/release advisory lock when creating partitions ([#2784](https://github.com/waku-org/nwaku/issues/2784)) ([c5d19c44](https://github.com/waku-org/nwaku/commit/c5d19c44)) @@ -48,6 +52,7 @@ Release highlights: - waku_discv5, peer_manager - add more logs help debug discovery issues ([#2705](https://github.com/waku-org/nwaku/issues/2705)) ([401630ee](https://github.com/waku-org/nwaku/commit/401630ee)) - Generic change to reduce the number of compilation warnings ([#2696](https://github.com/waku-org/nwaku/issues/2696)) ([78132dc1](https://github.com/waku-org/nwaku/commit/78132dc1)) + This release supports the following [libp2p protocols](https://docs.libp2p.io/concepts/protocols/): | Protocol | Spec status | Protocol id | | ---: | :---: | :--- | From e5bad0f2f10c367106bc421005e2afb1e3436aac Mon Sep 17 00:00:00 2001 From: gabrielmer <101006718+gabrielmer@users.noreply.github.com> Date: Mon, 17 Jun 2024 13:14:05 +0200 Subject: [PATCH 4/6] fix: revert "chore: adding observers for message logging (#2800)" (#2815) --- waku/node/waku_node.nim | 82 +++++++++--------------------------- waku/waku_relay/protocol.nim | 3 -- 2 files changed, 19 insertions(+), 66 deletions(-) diff --git a/waku/node/waku_node.nim b/waku/node/waku_node.nim index 31f571e8a..8d1ff8d79 100644 --- a/waku/node/waku_node.nim +++ b/waku/node/waku_node.nim @@ -225,6 +225,21 @@ proc registerRelayDefaultHandler(node: WakuNode, topic: PubsubTopic) = if node.wakuRelay.isSubscribed(topic): return + proc traceHandler(topic: PubsubTopic, msg: WakuMessage) {.async, gcsafe.} = + let msg_hash = topic.computeMessageHash(msg).to0xHex() + + notice "waku.relay received", + my_peer_id = node.peerId, + pubsubTopic = topic, + msg_hash = msg_hash, + receivedTime = getNowInNanosecondTime(), + payloadSizeBytes = msg.payload.len + + let msgSizeKB = msg.payload.len / 1000 + + waku_node_messages.inc(labelValues = ["relay"]) + waku_histogram_message_size.observe(msgSizeKB) + proc filterHandler(topic: PubsubTopic, msg: WakuMessage) {.async, gcsafe.} = if node.wakuFilter.isNil(): return @@ -240,6 +255,7 @@ proc registerRelayDefaultHandler(node: WakuNode, topic: PubsubTopic) = let defaultHandler = proc( topic: PubsubTopic, msg: WakuMessage ): Future[void] {.async, gcsafe.} = + await traceHandler(topic, msg) await filterHandler(topic, msg) await archiveHandler(topic, msg) @@ -376,61 +392,6 @@ proc startRelay*(node: WakuNode) {.async.} = info "relay started successfully" -proc generateRelayObserver(node: WakuNode): PubSubObserver = - proc logMessageInfo(peer: PubSubPeer, msgs: var RPCMsg, onRecv: bool) = - for msg in msgs.messages: - let msg_id = node.wakuRelay.msgIdProvider(msg).valueOr: - warn "Error generating message id", - my_peer_id = node.peerId, - from_peer_id = peer.peerId, - topic = msg.topic, - error = $error - continue - - let msg_id_short = shortLog(msg_id) - - let wakuMessage = WakuMessage.decode(msg.data).valueOr: - warn "Error decoding to Waku Message", - my_peer_id = node.peerId, - msg_id = msg_id_short, - from_peer_id = peer.peerId, - topic = msg.topic, - error = $error - continue - - let msg_hash = computeMessageHash(msg.topic, wakuMessage).to0xHex() - - if onRecv: - notice "received relay message", - my_peer_id = node.peerId, - msg_hash = msg_hash, - msg_id = msg_id_short, - from_peer_id = peer.peerId, - topic = msg.topic, - receivedTime = getNowInNanosecondTime(), - payloadSizeBytes = wakuMessage.payload.len - - let msgSizeKB = wakuMessage.payload.len / 1000 - waku_node_messages.inc(labelValues = ["relay"]) - waku_histogram_message_size.observe(msgSizeKB) - else: - notice "sent relay message", - my_peer_id = node.peerId, - msg_hash = msg_hash, - msg_id = msg_id_short, - to_peer_id = peer.peerId, - topic = msg.topic, - sentTime = getNowInNanosecondTime(), - payloadSizeBytes = wakuMessage.payload.len - - proc onRecv(peer: PubSubPeer, msgs: var RPCMsg) = - logMessageInfo(peer, msgs, onRecv = true) - - proc onSend(peer: PubSubPeer, msgs: var RPCMsg) = - discard - - return PubSubObserver(onRecv: onRecv, onSend: onSend) - proc mountRelay*( node: WakuNode, pubsubTopics: seq[string] = @[], @@ -451,11 +412,6 @@ proc mountRelay*( node.wakuRelay = initRes.value - # register relay observers for logging - debug "Registering Relay observers" - let observerLogger = node.generateRelayObserver() - node.wakuRelay.addObserver(observerLogger) - ## Add peer exchange handler if peerExchangeHandler.isSome(): node.wakuRelay.parameters.enablePX = true @@ -977,14 +933,14 @@ proc mountLightPush*( node: WakuNode, rateLimit: RateLimitSetting = DefaultGlobalNonRelayRateLimit ) {.async.} = info "mounting light push" - - var pushHandler = + + var pushHandler = if node.wakuRelay.isNil: debug "mounting lightpush without relay (nil)" getNilPushHandler() else: debug "mounting lightpush with relay" - let rlnPeer = + let rlnPeer = if isNil(node.wakuRlnRelay): debug "mounting lightpush without rln-relay" none(WakuRLNRelay) diff --git a/waku/waku_relay/protocol.nim b/waku/waku_relay/protocol.nim index ca59a6899..03d5b596e 100644 --- a/waku/waku_relay/protocol.nim +++ b/waku/waku_relay/protocol.nim @@ -180,9 +180,6 @@ proc addValidator*( ) {.gcsafe.} = w.wakuValidators.add((handler, errorMessage)) -proc addObserver*(w: WakuRelay, observer: PubSubObserver) {.gcsafe.} = - procCall GossipSub(w).addObserver(observer) - method start*(w: WakuRelay) {.async, base.} = debug "start" await procCall GossipSub(w).start() From 3f17b1d9bbc8937d1ea4c6714bce9a9018cabfc8 Mon Sep 17 00:00:00 2001 From: Ivan Folgueira Bande Date: Tue, 18 Jun 2024 18:45:05 +0200 Subject: [PATCH 5/6] update CHANGELOG.md to remove comment about observers for message log --- CHANGELOG.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f4d1ddcc3..f1007d481 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,7 +35,6 @@ Release highlights: ### Changes -- Adding observers for message logging ([#2800](https://github.com/waku-org/nwaku/issues/2800)) ([b5228652](https://github.com/waku-org/nwaku/commit/b5228652)) - Set msg_hash logs to notice level ([#2737](https://github.com/waku-org/nwaku/issues/2737)) ([f5d87c5b](https://github.com/waku-org/nwaku/commit/f5d87c5b)) - Minor enhancements ([#2789](https://github.com/waku-org/nwaku/issues/2789)) ([31bd6d71](https://github.com/waku-org/nwaku/commit/31bd6d71)) - postgres_driver - acquire/release advisory lock when creating partitions ([#2784](https://github.com/waku-org/nwaku/issues/2784)) ([c5d19c44](https://github.com/waku-org/nwaku/commit/c5d19c44)) From 88656c413184a5e3ebfca92882b2ced086d610bd Mon Sep 17 00:00:00 2001 From: Ivan FB <128452529+Ivansete-status@users.noreply.github.com> Date: Wed, 19 Jun 2024 21:59:53 +0200 Subject: [PATCH 6/6] CHANGELOG.md named sharding deprecation announcement Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com> --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f1007d481..cea6a11bb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ Notes: -* Start deprecate named sharding ([PR](https://github.com/waku-org/nwaku/pull/2723)) +* Named sharding will be deprecated in favor of static sharding. Topics in formats other than `/waku/2/rs//` will stop being supported starting from `v0.31.0` Release highlights: