DEBUG  tests.conftest:conftest.py:51 Running fixture setup: test_id DEBUG  tests.conftest:conftest.py:57 Running test: test_single_pubsub_topic[/waku/2/rs/2/0] with id: 2026-01-02_04-24-35__902ebc2c-94f3-4282-9ee6-4e2eeb934421 DEBUG  src.steps.common:common.py:19 Running fixture setup: common_setup DEBUG  src.steps.relay:relay.py:28 Running fixture setup: relay_setup DEBUG  src.steps.sharding:sharding.py:30 Running fixture setup: sharding_setup DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node1_2026-01-02_04-24-35__902ebc2c-94f3-4282-9ee6-4e2eeb934421__wakuorg_nwaku:latest.log DEBUG  src.node.waku_node:waku_node.py:90 Starting Node... DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.139.35 DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['40691', '40692', '40693', '40694', '40695'] DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2 DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN DEBUG  src.node.waku_node:waku_node.py:178 Using volumes [] DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 40691:40691 -p 40692:40692 -p 40693:40693 -p 40694:40694 -p 40695:40695 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=40693 --rest-port=40691 --tcp-port=40692 --discv5-udp-port=40694 --rest-address=0.0.0.0 --nat=extip:172.18.139.35 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=a80faddbb231a2f9455cc4d3bfa90b1fee6eaeb96bcb78596e4df06cbed5feb2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=40695 --metrics-logging=true --relay=true --filter=true DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.139.35 waku 2e3410b4ad4ce6775c67013223d6e97da13e0eb39eb148bcd4e4f34b0d2573e2 DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID 2e3410b4ad4c. Setting up logs at ./log/docker/node1_2026-01-02_04-24-35__902ebc2c-94f3-4282-9ee6-4e2eeb934421__wakuorg_nwaku:latest.log DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 40691 DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container fe84140e9c8f. Exiting log stream. ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container f40d389dcb3b. Exiting log stream. INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:40691/health" -H "Content-Type: application/json" -d 'None' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}' INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !! INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:40691/debug/v1/info" -H "Content-Type: application/json" -d 'None' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.139.35/tcp/40692/p2p/16Uiu2HAmUHFmrPZxz7GrbpBkMo5KCxXmT3sSzHR93zpPTh6tiW3f","/ip4/172.18.139.35/tcp/40693/ws/p2p/16Uiu2HAmUHFmrPZxz7GrbpBkMo5KCxXmT3sSzHR93zpPTh6tiW3f"],"enrUri":"enr:-L24QBLC5l4b_cCbbHZnqvhVqUk9T9D2tk-za_Xs8_gGrTjhF1sKp5StvTB3pbZhy9DM2l2lhRBgi2pPCoLu-l8fNtkCgmlkgnY0gmlwhKwSiyOKbXVsdGlhZGRyc5YACASsEosjBp70AAoErBKLIwae9d0DgnJzhQACAQAAiXNlY3AyNTZrMaED6CmnBOuyn6un1JNE5EORjvqbqd1Bli1teYro10Zib2aDdGNwgp70g3VkcIKe9oV3YWt1MgU"}' INFO  src.node.waku_node:waku_node.py:292 REST service is ready !! DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node2_2026-01-02_04-24-35__902ebc2c-94f3-4282-9ee6-4e2eeb934421__wakuorg_nwaku:latest.log DEBUG  src.node.waku_node:waku_node.py:90 Starting Node... DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.155.165 DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['6880', '6881', '6882', '6883', '6884'] DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2 DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN DEBUG  src.node.waku_node:waku_node.py:178 Using volumes [] DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 6880:6880 -p 6881:6881 -p 6882:6882 -p 6883:6883 -p 6884:6884 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6882 --rest-port=6880 --tcp-port=6881 --discv5-udp-port=6883 --rest-address=0.0.0.0 --nat=extip:172.18.155.165 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=eded68b7f2a9057a8e9eed9a1da6fc87faedfef8aaeaaa0fe76f47747eb3cdb4 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6884 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QBLC5l4b_cCbbHZnqvhVqUk9T9D2tk-za_Xs8_gGrTjhF1sKp5StvTB3pbZhy9DM2l2lhRBgi2pPCoLu-l8fNtkCgmlkgnY0gmlwhKwSiyOKbXVsdGlhZGRyc5YACASsEosjBp70AAoErBKLIwae9d0DgnJzhQACAQAAiXNlY3AyNTZrMaED6CmnBOuyn6un1JNE5EORjvqbqd1Bli1teYro10Zib2aDdGNwgp70g3VkcIKe9oV3YWt1MgU DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.155.165 waku f614929e948d6b050618430a6575ff95ed07ea284116d3fa57adfa9184968415 DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID f614929e948d. Setting up logs at ./log/docker/node2_2026-01-02_04-24-35__902ebc2c-94f3-4282-9ee6-4e2eeb934421__wakuorg_nwaku:latest.log DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 6880 DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:6880/health" -H "Content-Type: application/json" -d 'None' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}' INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !! INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:6880/debug/v1/info" -H "Content-Type: application/json" -d 'None' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.155.165/tcp/6881/p2p/16Uiu2HAkyErjyfEiwuQfYS43Mww1n361LjwMdVbzYYpwwesthD9V","/ip4/172.18.155.165/tcp/6882/ws/p2p/16Uiu2HAkyErjyfEiwuQfYS43Mww1n361LjwMdVbzYYpwwesthD9V"],"enrUri":"enr:-L24QLLf711yI8HpJ8PmCyD8Ap5MYO0bC2_gPfn6_qYnnhK8VKAu6YsASqW-Mn_L-NjVb1ymLP_E9qvHm_xz2vmvrYECgmlkgnY0gmlwhKwSm6WKbXVsdGlhZGRyc5YACASsEpulBhrhAAoErBKbpQYa4t0DgnJzhQACAQAAiXNlY3AyNTZrMaECOKmG9xCOHD_cuV1BiLeKi25Yrg_2S6NSz6DcPNecB4yDdGNwghrhg3VkcIIa44V3YWt1MgE"}' INFO  src.node.waku_node:waku_node.py:292 REST service is ready !! INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:6880/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.139.35/tcp/40692/p2p/16Uiu2HAmUHFmrPZxz7GrbpBkMo5KCxXmT3sSzHR93zpPTh6tiW3f"]' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK' INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:40691/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK' INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:6880/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK' INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:40691/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK' DEBUG  src.libs.common:common.py:47 Sleeping for 0.1 seconds DEBUG  src.steps.sharding:sharding.py:158 Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:40691/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767327878453360435,"ephemeral":false,"proof":""}]' DEBUG  src.steps.sharding:sharding.py:158 Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:6880/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None' INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1767327878453360435,"ephemeral":false,"proof":""}]' DEBUG  tests.conftest:conftest.py:59 Running fixture teardown: test_setup DEBUG  tests.conftest:conftest.py:83 Running fixture teardown: close_open_nodes DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id 2e3410b4ad4c DEBUG  src.node.waku_node:waku_node.py:241 Container stopped. DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id f614929e948d DEBUG  src.node.waku_node:waku_node.py:241 Container stopped. DEBUG  tests.conftest:conftest.py:98 Running fixture teardown: check_waku_log_errors DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs. DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.