271 lines
47 KiB
Plaintext

2026-03-08 04:33:51.759 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-08 04:33:51.759 DEBUG [tests.conftest] Running test: test_subscribe_and_publish_on_another_content_topic_from_another_shard with id: 2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5
2026-03-08 04:33:51.760 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-08 04:33:51.760 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-08 04:33:51.760 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-08 04:33:51.768 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:51.768 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5__wakuorg_nwaku:latest.log
2026-03-08 04:33:51.768 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:51.768 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:51.770 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:51.770 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.86.127
2026-03-08 04:33:51.770 DEBUG [src.node.docker_mananger] Generated ports ['58738', '58739', '58740', '58741', '58742']
2026-03-08 04:33:51.770 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:51.770 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:51.770 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:51.771 DEBUG [src.node.docker_mananger] docker run -i -t -p 58738:58738 -p 58739:58739 -p 58740:58740 -p 58741:58741 -p 58742:58742 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=58740 --rest-port=58738 --tcp-port=58739 --discv5-udp-port=58741 --rest-address=0.0.0.0 --nat=extip:172.18.86.127 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=0c79d1fbff6891a2e49b5bba4f6692fcc9fea020046cb8f3bdcfa5c031b1eaba --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=58742 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-08 04:33:51.965 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.86.127 waku aafef35a54069615a3018d405bf3633ca04d5d3896a8cbaedd29096312a7665a
2026-03-08 04:33:52.004 DEBUG [src.node.docker_mananger] Container started with ID aafef35a5406. Setting up logs at ./log/docker/node1_2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5__wakuorg_nwaku:latest.log
2026-03-08 04:33:52.004 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 58738
2026-03-08 04:33:52.004 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:33:52.567 ERROR [src.node.docker_mananger] Max retries reached for container 4e287c956a65. Exiting log stream.
2026-03-08 04:33:53.005 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58738/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:53.007 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:33:53.008 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:33:53.008 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58738/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:53.010 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.86.127/tcp/58739/p2p/16Uiu2HAm6rhfAKuyQf8qsMQNr6PA4NCdVJyJgnBJHb93KozVxkPj","/ip4/172.18.86.127/tcp/58740/ws/p2p/16Uiu2HAm6rhfAKuyQf8qsMQNr6PA4NCdVJyJgnBJHb93KozVxkPj"],"enrUri":"enr:-L24QAhONiGl9z2qWTL7Mkxzhx5TOoi1dhln3eFq7PNmZP1WErhZrmc57lvwB_xGyMG7qXx3yGstI8F2g0hf9DkA5NwCgmlkgnY0gmlwhKwSVn-KbXVsdGlhZGRyc5YACASsElZ_BuVzAAoErBJWfwbldN0DgnJzhQACAQAAiXNlY3AyNTZrMaECqdn5fzCLb-_xJvDbFIXpFnRtwY8O8Cv-yl5ixPtaHaqDdGNwguVzg3VkcILldYV3YWt1MgU"}'
2026-03-08 04:33:53.011 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:33:53.018 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:53.018 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5__wakuorg_nwaku:latest.log
2026-03-08 04:33:53.018 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:53.018 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:53.020 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:53.020 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.236.212
2026-03-08 04:33:53.020 DEBUG [src.node.docker_mananger] Generated ports ['22539', '22540', '22541', '22542', '22543']
2026-03-08 04:33:53.020 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:53.020 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:53.020 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:53.021 DEBUG [src.node.docker_mananger] docker run -i -t -p 22539:22539 -p 22540:22540 -p 22541:22541 -p 22542:22542 -p 22543:22543 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22541 --rest-port=22539 --tcp-port=22540 --discv5-udp-port=22542 --rest-address=0.0.0.0 --nat=extip:172.18.236.212 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=be3e939a45c35ff4e6dcbc5b5facdafc164a269dc8bdfeba4cef15ddceceb6d3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22543 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QAhONiGl9z2qWTL7Mkxzhx5TOoi1dhln3eFq7PNmZP1WErhZrmc57lvwB_xGyMG7qXx3yGstI8F2g0hf9DkA5NwCgmlkgnY0gmlwhKwSVn-KbXVsdGlhZGRyc5YACASsElZ_BuVzAAoErBJWfwbldN0DgnJzhQACAQAAiXNlY3AyNTZrMaECqdn5fzCLb-_xJvDbFIXpFnRtwY8O8Cv-yl5ixPtaHaqDdGNwguVzg3VkcILldYV3YWt1MgU --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-08 04:33:53.226 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.236.212 waku ab230ec4b575678ff0ecc7be42a192985e305bf68f9149d43c045b5105e35aba
2026-03-08 04:33:53.262 DEBUG [src.node.docker_mananger] Container started with ID ab230ec4b575. Setting up logs at ./log/docker/node2_2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5__wakuorg_nwaku:latest.log
2026-03-08 04:33:53.263 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22539
2026-03-08 04:33:53.263 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:33:54.264 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22539/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:54.281 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:54.381 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22539/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:54.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:33:54.384 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:33:54.385 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22539/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:54.387 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.236.212/tcp/22540/p2p/16Uiu2HAm3D2QkReNq6XiWKkCUh1TqFh9a1P8aEhniNuMoWnkHyis","/ip4/172.18.236.212/tcp/22541/ws/p2p/16Uiu2HAm3D2QkReNq6XiWKkCUh1TqFh9a1P8aEhniNuMoWnkHyis"],"enrUri":"enr:-L24QMdM6dkKjguU_TD16fOoE9ovUrz4G0e9HNkiOyDO3bOcCkweJ-IdAFlHfmtMDNODL6glH4n0YkiqTPFIxPsvHSACgmlkgnY0gmlwhKwS7NSKbXVsdGlhZGRyc5YACASsEuzUBlgMAAoErBLs1AZYDd0DgnJzhQACAQAAiXNlY3AyNTZrMaECc6ALpItV9fYNbOyJ7og-UhbLzeeXA12DNIAA_16lyuyDdGNwglgMg3VkcIJYDoV3YWt1MgE"}'
2026-03-08 04:33:54.387 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:33:54.388 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22539/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.86.127/tcp/58739/p2p/16Uiu2HAm6rhfAKuyQf8qsMQNr6PA4NCdVJyJgnBJHb93KozVxkPj"]'
2026-03-08 04:33:54.409 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:54.412 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58738/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-08 04:33:54.425 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:54.425 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22539/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-08 04:33:54.432 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:54.433 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58738/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:33:54.442 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:54.442 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:33:54.543 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:33:54.543 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58738/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:54.546 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1772944434433442017,"ephemeral":false,"proof":""}]'
2026-03-08 04:33:54.547 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-08 04:33:54.548 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22539/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:54.550 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1772944434433442017,"ephemeral":false,"proof":""}]'
2026-03-08 04:33:54.552 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58738/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:33:54.561 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:54.562 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:33:54.662 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:33:54.662 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58738/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:54.665 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772944434338986389,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772944434552366669,"ephemeral":false,"proof":""}]'
2026-03-08 04:33:54.710 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-08 04:33:54.711 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-08 04:33:54.711 DEBUG [src.node.waku_node] Stopping container with id aafef35a5406
2026-03-08 04:33:55.300 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:33:55.300 DEBUG [src.node.waku_node] Stopping container with id ab230ec4b575
2026-03-08 04:33:55.908 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:33:55.912 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-08 04:33:55.946 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:33:55.976 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:33:55.978 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-08 04:33:55.978 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-08 04:33:55.980 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5__wakuorg_nwaku:latest.log
2026-03-08 04:33:55.984 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-08_04-33-51__da52cf99-60fd-4c1c-8b50-b680208b2db5__wakuorg_nwaku:latest.log
2026-03-08 04:33:55.997 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-08 04:33:55.998 DEBUG [tests.conftest] Running test: test_subscribe_and_publish_on_another_content_topic_from_another_shard with id: 2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8
2026-03-08 04:33:55.999 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-08 04:33:56.000 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-08 04:33:56.000 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-08 04:33:56.012 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:56.012 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8__wakuorg_nwaku:latest.log
2026-03-08 04:33:56.013 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:56.013 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:56.015 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:56.015 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.24.237
2026-03-08 04:33:56.015 DEBUG [src.node.docker_mananger] Generated ports ['32120', '32121', '32122', '32123', '32124']
2026-03-08 04:33:56.016 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:56.016 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:56.016 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:56.016 DEBUG [src.node.docker_mananger] docker run -i -t -p 32120:32120 -p 32121:32121 -p 32122:32122 -p 32123:32123 -p 32124:32124 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=32122 --rest-port=32120 --tcp-port=32121 --discv5-udp-port=32123 --rest-address=0.0.0.0 --nat=extip:172.18.24.237 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=cee08e0bb7de7cf2ac0cbaed8ef2c88a4d8a841bcdd8d3c7a5f35e9da0bfb129 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=32124 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-08 04:33:56.180 ERROR [src.node.docker_mananger] Max retries reached for container aafef35a5406. Exiting log stream.
2026-03-08 04:33:56.230 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.24.237 waku 352297cccfbf4d967ae5d8c4b50a421cab6583d7dcfe30fe14f379f6d89ba10e
2026-03-08 04:33:56.270 DEBUG [src.node.docker_mananger] Container started with ID 352297cccfbf. Setting up logs at ./log/docker/node1_2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8__wakuorg_nwaku:latest.log
2026-03-08 04:33:56.270 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 32120
2026-03-08 04:33:56.271 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:33:56.908 ERROR [src.node.docker_mananger] Max retries reached for container ab230ec4b575. Exiting log stream.
2026-03-08 04:33:57.273 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32120/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:57.276 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:57.376 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32120/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:57.379 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:57.480 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32120/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:57.483 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:33:57.483 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:33:57.483 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32120/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:57.485 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.24.237/tcp/32121/p2p/16Uiu2HAmVRqXyvXA3xUpKkWc6J9JBaG511MVz8fMdXS8vCAMz8gR","/ip4/172.18.24.237/tcp/32122/ws/p2p/16Uiu2HAmVRqXyvXA3xUpKkWc6J9JBaG511MVz8fMdXS8vCAMz8gR"],"enrUri":"enr:-L24QCN5Btx6HAzMn9h48NyPz29lRcKIiaokg0G0vw9VjgXWFQI3bcpgK0Cy00Y4PEhSDMjAN25Lz7PoBywLf08l9QoCgmlkgnY0gmlwhKwSGO2KbXVsdGlhZGRyc5YACASsEhjtBn15AAoErBIY7QZ9et0DgnJzhQACAQAAiXNlY3AyNTZrMaED-TgoysY_5HjW8piDeB3y0MEdG1veIEG1mly71Mth2fKDdGNwgn15g3VkcIJ9e4V3YWt1MgU"}'
2026-03-08 04:33:57.486 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:33:57.493 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:57.493 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8__wakuorg_nwaku:latest.log
2026-03-08 04:33:57.493 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:57.493 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:57.495 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:57.495 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.84.181
2026-03-08 04:33:57.495 DEBUG [src.node.docker_mananger] Generated ports ['6613', '6614', '6615', '6616', '6617']
2026-03-08 04:33:57.495 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:57.495 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:57.495 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:57.496 DEBUG [src.node.docker_mananger] docker run -i -t -p 6613:6613 -p 6614:6614 -p 6615:6615 -p 6616:6616 -p 6617:6617 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6615 --rest-port=6613 --tcp-port=6614 --discv5-udp-port=6616 --rest-address=0.0.0.0 --nat=extip:172.18.84.181 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=bbbab7ec8eb25b7b81958c7b12fba7abfdca8c6e254eeab6d7a2fb0dadbce5cf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6617 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QCN5Btx6HAzMn9h48NyPz29lRcKIiaokg0G0vw9VjgXWFQI3bcpgK0Cy00Y4PEhSDMjAN25Lz7PoBywLf08l9QoCgmlkgnY0gmlwhKwSGO2KbXVsdGlhZGRyc5YACASsEhjtBn15AAoErBIY7QZ9et0DgnJzhQACAQAAiXNlY3AyNTZrMaED-TgoysY_5HjW8piDeB3y0MEdG1veIEG1mly71Mth2fKDdGNwgn15g3VkcIJ9e4V3YWt1MgU --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-08 04:33:57.700 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.84.181 waku c6fe4574fb83ae32e72ed0ffdce321b4ac60c57ac17f042e37e90deb536b0ba3
2026-03-08 04:33:57.739 DEBUG [src.node.docker_mananger] Container started with ID c6fe4574fb83. Setting up logs at ./log/docker/node2_2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8__wakuorg_nwaku:latest.log
2026-03-08 04:33:57.739 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 6613
2026-03-08 04:33:57.740 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:33:58.741 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:58.744 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:58.844 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:58.848 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:58.948 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:58.951 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:59.052 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.054 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:59.155 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.157 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:59.258 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.260 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:59.361 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.363 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:33:59.464 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.467 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:33:59.467 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:33:59.467 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.469 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.84.181/tcp/6614/p2p/16Uiu2HAm6hHmGGHkovSWFSfk4e9CrgPKVALvWa1t15fmJkBrMgMs","/ip4/172.18.84.181/tcp/6615/ws/p2p/16Uiu2HAm6hHmGGHkovSWFSfk4e9CrgPKVALvWa1t15fmJkBrMgMs"],"enrUri":"enr:-L24QCvbZJtSFnwEeL_2zxpDAcr-Hm7g9bkjU7HkH39m_pc9FM1vXreqc3dSsUFVJulk09nzd45TN7CIx_7zKr4LMccCgmlkgnY0gmlwhKwSVLWKbXVsdGlhZGRyc5YACASsElS1BhnWAAoErBJUtQYZ190DgnJzhQACAQAAiXNlY3AyNTZrMaECp3C6uY5261ECuo1D4ibENQupNVbxPykX5ydR2wTQz-aDdGNwghnWg3VkcIIZ2IV3YWt1MgE"}'
2026-03-08 04:33:59.470 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:33:59.470 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6613/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.24.237/tcp/32121/p2p/16Uiu2HAmVRqXyvXA3xUpKkWc6J9JBaG511MVz8fMdXS8vCAMz8gR"]'
2026-03-08 04:33:59.495 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:59.499 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:32120/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-08 04:33:59.507 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:59.508 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6613/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-08 04:33:59.512 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:59.513 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:32120/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:33:59.528 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:59.529 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:33:59.629 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:33:59.630 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32120/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.633 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1772944439513384756,"ephemeral":false,"proof":""}]'
2026-03-08 04:33:59.634 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-08 04:33:59.634 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6613/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.638 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1772944439513384756,"ephemeral":false,"proof":""}]'
2026-03-08 04:33:59.640 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:32120/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:33:59.649 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:59.650 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:33:59.750 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:33:59.750 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32120/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:59.753 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772944438211595987,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772944439640110175,"ephemeral":false,"proof":""}]'
2026-03-08 04:33:59.787 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-08 04:33:59.788 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-08 04:33:59.788 DEBUG [src.node.waku_node] Stopping container with id 352297cccfbf
2026-03-08 04:34:00.405 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:00.405 DEBUG [src.node.waku_node] Stopping container with id c6fe4574fb83
2026-03-08 04:34:00.995 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:00.997 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-08 04:34:01.039 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:01.068 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:01.070 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-08 04:34:01.070 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-08 04:34:01.070 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8__wakuorg_nwaku:latest.log
2026-03-08 04:34:01.072 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-08_04-33-55__4e7a1e71-4dc5-432e-8831-d22fd68ec9b8__wakuorg_nwaku:latest.log
2026-03-08 04:34:01.076 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-08 04:34:01.076 DEBUG [tests.conftest] Running test: test_subscribe_and_publish_on_another_content_topic_from_another_shard with id: 2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5
2026-03-08 04:34:01.077 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-08 04:34:01.077 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-08 04:34:01.077 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-08 04:34:01.085 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:34:01.085 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5__wakuorg_nwaku:latest.log
2026-03-08 04:34:01.085 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:34:01.085 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:34:01.087 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:34:01.087 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.104.11
2026-03-08 04:34:01.087 DEBUG [src.node.docker_mananger] Generated ports ['18682', '18683', '18684', '18685', '18686']
2026-03-08 04:34:01.087 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:34:01.087 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:34:01.087 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:34:01.088 DEBUG [src.node.docker_mananger] docker run -i -t -p 18682:18682 -p 18683:18683 -p 18684:18684 -p 18685:18685 -p 18686:18686 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=18684 --rest-port=18682 --tcp-port=18683 --discv5-udp-port=18685 --rest-address=0.0.0.0 --nat=extip:172.18.104.11 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f7c6ff5dad1e20e3c1a890a9f34c01edefbfa65dc0cea9aeea7b0bdabae9cd61 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=18686 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-08 04:34:01.272 ERROR [src.node.docker_mananger] Max retries reached for container 352297cccfbf. Exiting log stream.
2026-03-08 04:34:01.302 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.104.11 waku 80fce6aed9457cb0b1e21a0e1bd8c500f59b463846019ce2059e98d191bdf2a5
2026-03-08 04:34:01.336 DEBUG [src.node.docker_mananger] Container started with ID 80fce6aed945. Setting up logs at ./log/docker/node1_2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5__wakuorg_nwaku:latest.log
2026-03-08 04:34:01.339 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 18682
2026-03-08 04:34:01.339 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:34:01.844 ERROR [src.node.docker_mananger] Max retries reached for container c6fe4574fb83. Exiting log stream.
2026-03-08 04:34:02.340 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18682/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.342 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:34:02.343 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:34:02.343 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18682/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.345 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.104.11/tcp/18683/p2p/16Uiu2HAmSxbr4YYCcE9bkUqDNGkYQ1VGCe8LvKDrAaocB79HkWu4","/ip4/172.18.104.11/tcp/18684/ws/p2p/16Uiu2HAmSxbr4YYCcE9bkUqDNGkYQ1VGCe8LvKDrAaocB79HkWu4"],"enrUri":"enr:-L24QC2NCtxqMkhZIWGMO2M8T57srG0fwREH_70Tt004HUd1M06Lc-g-21xEDE3AwDWIGOZrbMUp_8aOANqjrc_-J60CgmlkgnY0gmlwhKwSaAuKbXVsdGlhZGRyc5YACASsEmgLBkj7AAoErBJoCwZI_N0DgnJzhQACAQAAiXNlY3AyNTZrMaED1IagUmLjhvSoiV6qLTRof6lnCn1Ev9hO6W3LtLwDpReDdGNwgkj7g3VkcIJI_YV3YWt1MgU"}'
2026-03-08 04:34:02.345 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:34:02.353 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:34:02.353 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5__wakuorg_nwaku:latest.log
2026-03-08 04:34:02.353 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:34:02.353 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:34:02.355 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:34:02.355 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.5.80
2026-03-08 04:34:02.355 DEBUG [src.node.docker_mananger] Generated ports ['40811', '40812', '40813', '40814', '40815']
2026-03-08 04:34:02.355 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:34:02.355 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:34:02.355 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:34:02.356 DEBUG [src.node.docker_mananger] docker run -i -t -p 40811:40811 -p 40812:40812 -p 40813:40813 -p 40814:40814 -p 40815:40815 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=40813 --rest-port=40811 --tcp-port=40812 --discv5-udp-port=40814 --rest-address=0.0.0.0 --nat=extip:172.18.5.80 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=47dfbc3348f8732d28ca8a2b6c45892d43f9c0289366c62a1b65ee98a66dccfb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=40815 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC2NCtxqMkhZIWGMO2M8T57srG0fwREH_70Tt004HUd1M06Lc-g-21xEDE3AwDWIGOZrbMUp_8aOANqjrc_-J60CgmlkgnY0gmlwhKwSaAuKbXVsdGlhZGRyc5YACASsEmgLBkj7AAoErBJoCwZI_N0DgnJzhQACAQAAiXNlY3AyNTZrMaED1IagUmLjhvSoiV6qLTRof6lnCn1Ev9hO6W3LtLwDpReDdGNwgkj7g3VkcIJI_YV3YWt1MgU --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-08 04:34:02.565 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.5.80 waku 843047f6da1a87790a553d11efca352987d4e1c12b2d62d3c5388e83bc605a22
2026-03-08 04:34:02.599 DEBUG [src.node.docker_mananger] Container started with ID 843047f6da1a. Setting up logs at ./log/docker/node2_2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5__wakuorg_nwaku:latest.log
2026-03-08 04:34:02.599 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 40811
2026-03-08 04:34:02.600 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:34:03.601 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40811/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.609 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-08 04:34:03.709 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40811/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.712 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:34:03.713 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:34:03.713 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40811/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.716 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.5.80/tcp/40812/p2p/16Uiu2HAkwjnW6NnQYu878nEu8xVoAjxWXMRHNJf6hVndpWzbikzu","/ip4/172.18.5.80/tcp/40813/ws/p2p/16Uiu2HAkwjnW6NnQYu878nEu8xVoAjxWXMRHNJf6hVndpWzbikzu"],"enrUri":"enr:-L24QH0PSZqyq8qmsdbPZ0dvgz8kK-REBwUg_nxxflpM0RBDDLGx9qaOapEKzlQAatW0hrFqoYNEJwdsd3Gzaax8j_QCgmlkgnY0gmlwhKwSBVCKbXVsdGlhZGRyc5YACASsEgVQBp9sAAoErBIFUAafbd0DgnJzhQACAQAAiXNlY3AyNTZrMaECIls0DZj-XfhnT2ymIrwosLa7eOUjz1of3rVFyueX-BKDdGNwgp9sg3VkcIKfboV3YWt1MgE"}'
2026-03-08 04:34:03.716 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:34:03.719 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40811/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.104.11/tcp/18683/p2p/16Uiu2HAmSxbr4YYCcE9bkUqDNGkYQ1VGCe8LvKDrAaocB79HkWu4"]'
2026-03-08 04:34:03.744 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:03.747 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18682/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-08 04:34:03.754 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:03.754 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40811/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-08 04:34:03.763 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:03.764 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18682/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:34:03.779 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:03.781 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:34:03.882 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.882 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18682/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.885 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1772944443764682694,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.887 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.888 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40811/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.891 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1772944443764682694,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.893 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18682/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:34:03.903 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:03.904 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:34:04.004 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:04.005 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18682/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:04.009 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772944442577911942,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772944443892878612,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:04.019 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-08 04:34:04.020 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-08 04:34:04.020 DEBUG [src.node.waku_node] Stopping container with id 80fce6aed945
2026-03-08 04:34:04.610 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:04.610 DEBUG [src.node.waku_node] Stopping container with id 843047f6da1a
2026-03-08 04:34:05.210 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:05.214 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-08 04:34:05.276 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:05.305 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:05.306 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-08 04:34:05.306 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-08 04:34:05.306 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5__wakuorg_nwaku:latest.log
2026-03-08 04:34:05.308 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-08_04-34-01__c782dfe7-8ed8-4e6f-81bd-f625b9be4aa5__wakuorg_nwaku:latest.log