195 lines
32 KiB
Plaintext

2026-03-13 04:41:20.262 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:41:20.263 DEBUG [tests.conftest] Running test: test_cant_publish_on_not_subscribed_shard with id: 2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b
2026-03-13 04:41:20.264 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:41:20.266 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:41:20.266 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-13 04:41:20.278 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:41:20.279 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b__wakuorg_nwaku:latest.log
2026-03-13 04:41:20.279 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:41:20.279 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:41:20.282 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:41:20.283 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.114.2
2026-03-13 04:41:20.283 DEBUG [src.node.docker_mananger] Generated ports ['34924', '34925', '34926', '34927', '34928']
2026-03-13 04:41:20.283 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-13 04:41:20.284 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-13 04:41:20.285 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:41:20.285 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:41:20.285 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:41:20.286 DEBUG [src.node.docker_mananger] docker run -i -t -p 34924:34924 -p 34925:34925 -p 34926:34926 -p 34927:34927 -p 34928:34928 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=34926 --rest-port=34924 --tcp-port=34925 --discv5-udp-port=34927 --rest-address=0.0.0.0 --nat=extip:172.18.114.2 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=1c9bc92e2e66fd58a58aefd3650f0bbd7659b1b19aaccf609bdb65ac2cee062c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=34928 --metrics-logging=true --relay=true --filter=true
2026-03-13 04:41:20.341 ERROR [src.node.docker_mananger] Max retries reached for container b89a603bcf95. Exiting log stream.
2026-03-13 04:41:20.493 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.114.2 waku c2a9f89c3db71d0cdc39694fc088aa80c9a7fef1b8954a06dafad884a51e786b
2026-03-13 04:41:20.529 DEBUG [src.node.docker_mananger] Container started with ID c2a9f89c3db7. Setting up logs at ./log/docker/node1_2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b__wakuorg_nwaku:latest.log
2026-03-13 04:41:20.530 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 34924
2026-03-13 04:41:20.531 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:41:21.180 ERROR [src.node.docker_mananger] Max retries reached for container a11a8c261b78. Exiting log stream.
2026-03-13 04:41:21.532 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:21.535 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:21.635 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:21.638 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:21.739 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:21.742 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:21.843 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:21.845 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:21.946 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:21.949 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:41:21.949 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:41:21.949 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:21.952 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.114.2/tcp/34925/p2p/16Uiu2HAmBAikVfNHfe5kwk87QnSPgCZpkoTiZyEtCRU4vSKVKwHE","/ip4/172.18.114.2/tcp/34926/ws/p2p/16Uiu2HAmBAikVfNHfe5kwk87QnSPgCZpkoTiZyEtCRU4vSKVKwHE"],"enrUri":"enr:-L24QD8SlkJAHiO9SwLlAFLC6omGDWqntXv0U8mjkjHagIhJAiNhZs6P93SlfVQ5S1iKBqO0lr7vN-BdtrbiqhSRycQCgmlkgnY0gmlwhKwScgKKbXVsdGlhZGRyc5YACASsEnICBohtAAoErBJyAgaIbt0DgnJzhQACAQAAiXNlY3AyNTZrMaEC6eZpWt6-GrWKT4hLmgTlQ5ndj-yg_X5mEFny0XyVRNWDdGNwgohtg3VkcIKIb4V3YWt1MgU"}'
2026-03-13 04:41:21.952 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:41:21.960 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:41:21.960 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b__wakuorg_nwaku:latest.log
2026-03-13 04:41:21.960 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:41:21.960 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:41:21.961 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:41:21.962 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.106.30
2026-03-13 04:41:21.962 DEBUG [src.node.docker_mananger] Generated ports ['26181', '26182', '26183', '26184', '26185']
2026-03-13 04:41:21.962 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-13 04:41:21.962 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-13 04:41:21.962 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:41:21.962 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:41:21.962 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:41:21.963 DEBUG [src.node.docker_mananger] docker run -i -t -p 26181:26181 -p 26182:26182 -p 26183:26183 -p 26184:26184 -p 26185:26185 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=26183 --rest-port=26181 --tcp-port=26182 --discv5-udp-port=26184 --rest-address=0.0.0.0 --nat=extip:172.18.106.30 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=903abcca5e997ffa1f0ed9ef86c1d2ffebabc46e8e0e9b7afcb63885dc8b10c8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=26185 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QD8SlkJAHiO9SwLlAFLC6omGDWqntXv0U8mjkjHagIhJAiNhZs6P93SlfVQ5S1iKBqO0lr7vN-BdtrbiqhSRycQCgmlkgnY0gmlwhKwScgKKbXVsdGlhZGRyc5YACASsEnICBohtAAoErBJyAgaIbt0DgnJzhQACAQAAiXNlY3AyNTZrMaEC6eZpWt6-GrWKT4hLmgTlQ5ndj-yg_X5mEFny0XyVRNWDdGNwgohtg3VkcIKIb4V3YWt1MgU
2026-03-13 04:41:22.165 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.106.30 waku 753034162ae6551c427c1578681863f42cd9ecf954110c32615a50ae96a178c0
2026-03-13 04:41:22.201 DEBUG [src.node.docker_mananger] Container started with ID 753034162ae6. Setting up logs at ./log/docker/node2_2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b__wakuorg_nwaku:latest.log
2026-03-13 04:41:22.202 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 26181
2026-03-13 04:41:22.202 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:41:23.202 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26181/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:23.225 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:23.325 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26181/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:23.328 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:41:23.328 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:41:23.328 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26181/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:23.331 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.106.30/tcp/26182/p2p/16Uiu2HAmHuLehH88Rr7AVHtv7vJGik3GXBP56UZxZDZFNH3kd5SU","/ip4/172.18.106.30/tcp/26183/ws/p2p/16Uiu2HAmHuLehH88Rr7AVHtv7vJGik3GXBP56UZxZDZFNH3kd5SU"],"enrUri":"enr:-L24QO5IR36RVL-sxwlk1GxIrvcF0qv70loCGXQUrrvWZSkpXwGrw10klwdA3AfapCqPtaQUAj-7FWWmLXKYRQ66f5wCgmlkgnY0gmlwhKwSah6KbXVsdGlhZGRyc5YACASsEmoeBmZGAAoErBJqHgZmR90DgnJzhQACAQAAiXNlY3AyNTZrMaEDTfeE8-1atuN-xthJV2nMwLJ4RbOTBWwxkSjNg-qJs2WDdGNwgmZGg3VkcIJmSIV3YWt1MgE"}'
2026-03-13 04:41:23.331 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:41:23.332 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:26181/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.114.2/tcp/34925/p2p/16Uiu2HAmBAikVfNHfe5kwk87QnSPgCZpkoTiZyEtCRU4vSKVKwHE"]'
2026-03-13 04:41:23.361 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:23.363 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:34924/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-13 04:41:23.373 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:23.374 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:26181/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-13 04:41:23.379 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:23.381 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:34924/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:41:23.392 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:23.393 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-13 04:41:23.493 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-13 04:41:23.494 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:34924/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:23.497 ERROR [src.node.api_clients.base_client] HTTP error occurred: 404 Client Error: Not Found for url: http://127.0.0.1:34924/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1. Response content: b''
2026-03-13 04:41:23.574 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:41:23.575 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:41:23.575 DEBUG [src.node.waku_node] Stopping container with id c2a9f89c3db7
2026-03-13 04:41:24.192 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:41:24.192 DEBUG [src.node.waku_node] Stopping container with id 753034162ae6
2026-03-13 04:41:24.796 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:41:24.798 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:41:24.834 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:41:24.863 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:41:24.864 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-13 04:41:24.865 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-13 04:41:24.866 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b__wakuorg_nwaku:latest.log
2026-03-13 04:41:24.867 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-13_04-41-20__7b06411f-9871-4bfa-9c7c-0ff2f17afc1b__wakuorg_nwaku:latest.log
2026-03-13 04:41:24.871 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:41:24.871 DEBUG [tests.conftest] Running test: test_cant_publish_on_not_subscribed_shard with id: 2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871
2026-03-13 04:41:24.872 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:41:24.872 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:41:24.872 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-13 04:41:24.879 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:41:24.880 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871__wakuorg_nwaku:latest.log
2026-03-13 04:41:24.880 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:41:24.880 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:41:24.881 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:41:24.881 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.99.15
2026-03-13 04:41:24.882 DEBUG [src.node.docker_mananger] Generated ports ['28883', '28884', '28885', '28886', '28887']
2026-03-13 04:41:24.882 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-13 04:41:24.882 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-13 04:41:24.882 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:41:24.882 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:41:24.882 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:41:24.882 DEBUG [src.node.docker_mananger] docker run -i -t -p 28883:28883 -p 28884:28884 -p 28885:28885 -p 28886:28886 -p 28887:28887 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=28885 --rest-port=28883 --tcp-port=28884 --discv5-udp-port=28886 --rest-address=0.0.0.0 --nat=extip:172.18.99.15 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=ba6bbeb3c82619eff3b4ebcf8d10aa8b8fa6ca2bef958cc5deb2bb47bc6071e8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=28887 --metrics-logging=true --relay=true --filter=true
2026-03-13 04:41:25.068 ERROR [src.node.docker_mananger] Max retries reached for container c2a9f89c3db7. Exiting log stream.
2026-03-13 04:41:25.079 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.99.15 waku c3ae7d66b8176a75f706c973de64bfa58d19f0eb24ea52e9736639acfffa1a73
2026-03-13 04:41:25.119 DEBUG [src.node.docker_mananger] Container started with ID c3ae7d66b817. Setting up logs at ./log/docker/node1_2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871__wakuorg_nwaku:latest.log
2026-03-13 04:41:25.120 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 28883
2026-03-13 04:41:25.121 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:41:25.673 ERROR [src.node.docker_mananger] Max retries reached for container 753034162ae6. Exiting log stream.
2026-03-13 04:41:26.121 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.124 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.225 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.227 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.327 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.330 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.430 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.433 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.533 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.536 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.636 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.639 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.739 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.741 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.842 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.844 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:26.945 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:26.947 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:27.047 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:27.050 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:27.150 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:27.153 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:27.253 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:27.300 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:27.400 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:27.403 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:41:27.403 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:41:27.403 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:27.405 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.99.15/tcp/28884/p2p/16Uiu2HAmHNKGvG48n4B8aDTgZhKMAVKHAQ71qP4FMAbjKzw5AQNH","/ip4/172.18.99.15/tcp/28885/ws/p2p/16Uiu2HAmHNKGvG48n4B8aDTgZhKMAVKHAQ71qP4FMAbjKzw5AQNH"],"enrUri":"enr:-L24QN27vtZrDAc0wmqtgC7UHtTxBsUP_G50lo_whJ3Z2zdhHWvBTEgezoQx82ogBLGiKtwdAHFSWMipOJTPMR1orAwCgmlkgnY0gmlwhKwSYw-KbXVsdGlhZGRyc5YACASsEmMPBnDUAAoErBJjDwZw1d0DgnJzhQACAQAAiXNlY3AyNTZrMaEDRgT1g4GhhhQk0klUsx2x1ft4rWibMHLptPmrpCkiRbaDdGNwgnDUg3VkcIJw1oV3YWt1MgU"}'
2026-03-13 04:41:27.406 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:41:27.413 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:41:27.413 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871__wakuorg_nwaku:latest.log
2026-03-13 04:41:27.413 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:41:27.413 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:41:27.415 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:41:27.415 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.33.69
2026-03-13 04:41:27.415 DEBUG [src.node.docker_mananger] Generated ports ['50292', '50293', '50294', '50295', '50296']
2026-03-13 04:41:27.415 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-13 04:41:27.415 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-13 04:41:27.416 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:41:27.416 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:41:27.416 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:41:27.416 DEBUG [src.node.docker_mananger] docker run -i -t -p 50292:50292 -p 50293:50293 -p 50294:50294 -p 50295:50295 -p 50296:50296 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=50294 --rest-port=50292 --tcp-port=50293 --discv5-udp-port=50295 --rest-address=0.0.0.0 --nat=extip:172.18.33.69 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=66c472bbaddba6436ad66a7fd7ea08f655ae4ec9bfac55dc4df88c35b3acffa6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=50296 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QN27vtZrDAc0wmqtgC7UHtTxBsUP_G50lo_whJ3Z2zdhHWvBTEgezoQx82ogBLGiKtwdAHFSWMipOJTPMR1orAwCgmlkgnY0gmlwhKwSYw-KbXVsdGlhZGRyc5YACASsEmMPBnDUAAoErBJjDwZw1d0DgnJzhQACAQAAiXNlY3AyNTZrMaEDRgT1g4GhhhQk0klUsx2x1ft4rWibMHLptPmrpCkiRbaDdGNwgnDUg3VkcIJw1oV3YWt1MgU
2026-03-13 04:41:27.620 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.33.69 waku 51194fff68425cd0093dd18ff4df1a967dfbe173501aac5cc7f823eac1f54519
2026-03-13 04:41:27.655 DEBUG [src.node.docker_mananger] Container started with ID 51194fff6842. Setting up logs at ./log/docker/node2_2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871__wakuorg_nwaku:latest.log
2026-03-13 04:41:27.656 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 50292
2026-03-13 04:41:27.656 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:41:28.656 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50292/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:28.659 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:28.759 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50292/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:28.762 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-13 04:41:28.862 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50292/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:28.865 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:41:28.865 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:41:28.865 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50292/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:28.867 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.33.69/tcp/50293/p2p/16Uiu2HAmVd8MkhAGKZTphQSpCDkSwgAHgUzAwuTEvcq1nq9qQXMx","/ip4/172.18.33.69/tcp/50294/ws/p2p/16Uiu2HAmVd8MkhAGKZTphQSpCDkSwgAHgUzAwuTEvcq1nq9qQXMx"],"enrUri":"enr:-L24QCGJkoC7hPsRilXiT27rKdoh6ckDFSA6P9OASGzp-m8XLR4Bn-JgRLmEDjOlYprxDkNFaQmz-looX0AR1IOr-yYCgmlkgnY0gmlwhKwSIUWKbXVsdGlhZGRyc5YACASsEiFFBsR1AAoErBIhRQbEdt0DgnJzhQACAQAAiXNlY3AyNTZrMaED_BySII91Q8Rlk-jJD_qWDD5i31A5Le0ie8RA3oEP8Q-DdGNwgsR1g3VkcILEd4V3YWt1MgE"}'
2026-03-13 04:41:28.867 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:41:28.868 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:50292/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.99.15/tcp/28884/p2p/16Uiu2HAmHNKGvG48n4B8aDTgZhKMAVKHAQ71qP4FMAbjKzw5AQNH"]'
2026-03-13 04:41:28.891 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:28.893 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28883/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-13 04:41:28.902 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:28.902 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:50292/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-13 04:41:28.910 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:28.911 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:28883/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:41:28.921 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:41:28.923 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-13 04:41:29.023 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-13 04:41:29.024 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:28883/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:41:29.026 ERROR [src.node.api_clients.base_client] HTTP error occurred: 404 Client Error: Not Found for url: http://127.0.0.1:28883/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F1. Response content: b''
2026-03-13 04:41:29.067 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:41:29.068 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:41:29.068 DEBUG [src.node.waku_node] Stopping container with id c3ae7d66b817
2026-03-13 04:41:29.648 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:41:29.649 DEBUG [src.node.waku_node] Stopping container with id 51194fff6842
2026-03-13 04:41:30.253 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:41:30.255 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:41:30.281 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:41:30.307 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:41:30.309 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-13 04:41:30.310 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-13 04:41:30.314 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871__wakuorg_nwaku:latest.log
2026-03-13 04:41:30.316 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-13_04-41-24__e4ea7e51-dc0f-4047-a861-553aaeb3f871__wakuorg_nwaku:latest.log