diff --git a/content/vac/qa/g/waku/interop-testing-02.md b/content/vac/qa/g/waku/interop-testing-02.md new file mode 100644 index 000000000..42a7d9df9 --- /dev/null +++ b/content/vac/qa/g/waku/interop-testing-02.md @@ -0,0 +1,45 @@ +--- +title: "Interop Testing Part 2" +--- +## `vac:qa::waku:interop-testing-02` +--- + +```mermaid +%%{ + init: { + 'theme': 'base', + 'themeVariables': { + 'primaryColor': '#BB2528', + 'primaryTextColor': '#fff', + 'primaryBorderColor': '#7C0000', + 'lineColor': '#F8B229', + 'secondaryColor': '#006100', + 'tertiaryColor': '#fff' + } + } +}%% +gantt + tickInterval 1month + dateFormat YYYY-MM-DD + section Status + Interop Testing Part 2: 2024-07-01, 2024-12-31 +``` + +- status: 0% +- CC: Florin, Roman + +### Description + +Add new coverage for the interop testing framework + +* peer exchange +* discv5 +* peer & connection management +* edge cases +* more complex scenario +* create tests for known issues found by devs / node operators / integrators + +### Justification + + +### Deliverables diff --git a/content/vac/qa/g/waku/interop-testing.md b/content/vac/qa/g/waku/interop-testing.md index 380cedf06..b5cc4f52a 100644 --- a/content/vac/qa/g/waku/interop-testing.md +++ b/content/vac/qa/g/waku/interop-testing.md @@ -25,19 +25,17 @@ gantt Interop Testing: 2023-09-01, 2024-04-30 ``` -- status: 30% +- status: 100% - CC: Florin, Roman ### Description +Create an interop testing framework that can run Waku nodes and cover it's + * filter * lightpush -* store +* store (incl. store v3) * relay -* peer exchange -* discv5 -* peer & connection management -* filter nwaku service node + js waku client node * nwaku <> go-waku interop * ci integrations * nightly reports @@ -47,5 +45,76 @@ gantt ### Deliverables - - +* Created scalable pytest framework +* Reached ~ 320 tests +* Created CI github actions workflows that: + * Runs nightly on latest nwaku and go-waku versions + * Runs tests between multiple nwaku nodes but with nwaku as service node and go-waku as client nodes in an interop way + * Can be run on demand against any nwaku / go-waku version + * Publishes nice reports with history and logs using github pages, [check example](https://waku-org.github.io/waku-interop-tests/nim/210/) + * Notifies waku discord channel about results of tests and pings developers if there are failures +* Coverage: + * [filter](https://github.com/waku-org/waku-interop-tests/tree/master/tests/filter) + * [lightpush](https://github.com/waku-org/waku-interop-tests/tree/master/tests/light_push) + * [relay](https://github.com/waku-org/waku-interop-tests/tree/master/tests/relay) + * [store](https://github.com/waku-org/waku-interop-tests/tree/master/tests/store) + * [sharding](https://github.com/waku-org/waku-interop-tests/tree/master/tests/sharding) +* Nwaku issues found: + * 2719: store v3 response format issues + * 2717: nwaku crashes for a store v3 request with invalid cursor format + * 2716: passing a cursor that doesn't correspond to any message in the store will return all messages + * 2715: store v3 returns error "waku message hash parsing error: Incorrect base64 string" for some cursors + * 2644: nwaku node fails to start without a shard flag + * 2586: node doesn't store messages if relay is disabled + * 2582: contentTopic naming not consistent in the store response where is's content_topic + * 2567: lightpush fails with Failed to request a message push: dial_failure after the peer node restart + * 2565: strange errors when light pushing messages with payload >= 300 kb + * 2552: node ca be started on multiple clusters + * 2550: node crashes with Message: AsyncEventQueue size exceeds limits when there are many flags to the docker start command + * 2546: only receive messages if someone subscribes explicitly via REST API to a pubsubTopic + * 2538: autosharding resolves content topics to wrong shard + * 2512: some lightnodes are not receiving filter push in certain conditions + * 2437: relay publish fails with 400 Bad Request when message contains an unknown field + * 2436: relay publish fails with 400 Bad Request when message contains ephemeral field + * 2388: relayed messages reach recently started peer with a big delay (~60 seconds) + * 2380: Relayed messages are not stored when running nwaku with docker compose + * 2372: failed to setup archive driver: Postgres has been configured but not been compiled. Check compiler definitions + * 2371: multiple messages published in the same second via RLN RELAY are not dropped + * 2320: Filter relay/v1/messages GET returns duplicate messages + * 2319: Relay publish returns Failed to publish: timedout when a peer filter node is disconnected + * 2315: updating a non existing subscriptions returns no error + * 2299: Relay connection works no more + * 2286: filter/v2/subscriptions response not in the expected format + * 2255: pubsub topic not mandatory for filter/v2/subscriptions + * 2214: relay publish fails with 400 Bad Request when message contains meta field + * 2198: relay push with malformed timestamp crashes nwaku + * 2147: store query cursor misbehaving for specific cursors + * 2117: store response is empty when requests contains invalid cursor +* Go-Waku issues found: + * 1110: store v3 - passing a cursor that doesn't correspond to any message in the store will return all messages + * 1109: store v3 returns error "illegal base64 data at input byte" for some cursors/hashes + * 1108: pubsubTopic and contentTopics are required for store v3 requests + * 1107: standardize store types by using camel case instead of snake case + * 1106: store v3 local queries time out + * 1087: failed to negotiate protocol: protocols not supported: [/vac/waku/store/2.0.0-beta4] when the peer node has store disabled + * 1079: missing RequestId field error when lightpush has unexpected payload of content topic + * 1078: lightpush on non subscribed pubsub topic hangs + * 1076: strange errors when light pushing messages with payload >= 300 kb + * 1074: all REST API calls return 200 with empty response + * 1068: ephemeral field is ignored and not returned when retrieving messages even if the message contained this field + * 1064: subscription not found if we start the node with the --pubsub-topic and we attempt to retrieve messages + * 1061: dont harcode clusterid for autosharding + * 1054: filter subscribe on a pubsubtopic from a different cluster id freezes + * 1034: DELETE /relay/v1/subscriptions freezes in certain conditions + * 988: invalid memory address or nil pointer dereference when trying to connect nodes + * 972: filter/v2/subscriptions take a lot of time and even timeout sometimes + * 971: Unsubscribing from one pubsubtopic seems to unsubscribe from all + * 970: ping request failed with 503 when peer has no subscription + * 969: PUT /filter/v2/subscriptions doesn't exist + * 968: 503 instead of 400 when a filter/v2/subscriptions bad request is sent + * 967: filter/v2/subscriptions freezes when pubsubtopic is from the non-default (0) cluster + * 960: Strange "not subscribed to pubsubTopic" errors for filter/v2/messages GET requests + * 928: encoding/hex: odd length hex string for filter/v2/subscriptions POST requests + * 923: discv5/discover.go messages flooding the docker DEBUG logs + * 922: duplicate validator for topic error when trying to re-subscribe to previously unsubscribed topic + * 914: REST relay publish returns HTTP 500 Internal Server Error instead of 4XX for invalid requests E:REST API service node diff --git a/content/vac/qa/index.md b/content/vac/qa/index.md index a54f210a4..4dc77fb16 100644 --- a/content/vac/qa/index.md +++ b/content/vac/qa/index.md @@ -4,7 +4,7 @@ tags: - dst - vac date: 2024-02-06 -lastmod: 2024-05-13 +lastmod: 2024-05-27 --- ## `vac:qa::` @@ -19,12 +19,13 @@ lastmod: 2024-05-13 * [[vac/qa/g/waku/test-automation-rln|test-automation-rln ]] * [x] [[vac/qa/g/waku/test-automation-sharding|test-automation-sharding ]] * [x] [[vac/qa/g/waku/test-automation-go-waku|test-automation-go-waku ]] -* [[vac/qa/g/waku/interop-testing|interop-testing ]] +* [x] [[vac/qa/g/waku/interop-testing|interop-testing ]] +* [[vac/qa/g/waku/interop-testing-02|interop-testing-02 ]] * [[vac/qa/g/waku/maintenance-js-waku|maintenance-js-waku ]] * [[vac/qa/g/waku/maintenance-nwaku|maintenance-nwaku ]] * [[vac/qa/g/waku/maintenance-go-waku|maintenance-go-nwaku ]] * [[vac/qa/g/waku/ws-stress-testing|ws-stress-testing ]] -* [[vac/qa/g/waku/status-go-cli-testing|status-go-cli-testing ]] +* [[vac/qa/g/waku/test-automation-status-go-cli|test-automation-status-go-cli ]] ### `vac:` * [[vac/qa/g/vac/test-automation-nim-libp2p|test-automation-nim-libp2p ]]