272 lines
52 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

DEBUG  tests.conftest:conftest.py:51 Running fixture setup: test_id
DEBUG  tests.conftest:conftest.py:57 Running test: test_start_node_with_50_pubsub_topics with id: 2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0
DEBUG  src.steps.common:common.py:19 Running fixture setup: common_setup
DEBUG  src.steps.relay:relay.py:28 Running fixture setup: relay_setup
DEBUG  src.steps.sharding:sharding.py:30 Running fixture setup: sharding_setup
DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest
DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node1_2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:90 Starting Node...
DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku
DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists
DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.19.184
DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['4624', '4625', '4626', '4627', '4628']
DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2
DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args
DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set
INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN
DEBUG  src.node.waku_node:waku_node.py:178 Using volumes []
DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 4624:4624 -p 4625:4625 -p 4626:4626 -p 4627:4627 -p 4628:4628 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4626 --rest-port=4624 --tcp-port=4625 --discv5-udp-port=4627 --rest-address=0.0.0.0 --nat=extip:172.18.19.184 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=2917d50fc4789ef5942cddc3efec3bd560fe9a3aac4a098ed738e737f0f155a2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4628 --metrics-logging=true --relay=true --filter=true
ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container dce4ee28520e. Exiting log stream.
DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.19.184 waku 26fedc537f6ca3b22db43a6ad0f06ef7f86c6b4054c37a060f20e2afaeb36111
DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID 26fedc537f6c. Setting up logs at ./log/docker/node1_2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 4624
DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds
ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container 59c99c037499. Exiting log stream.
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:4624/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:4624/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:4624/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:4624/debug/v1/info" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.19.184/tcp/4625/p2p/16Uiu2HAm7rADiYA1qTp963e1m51FsA2nKhnr9TeUFiBr6xbo6gSr","/ip4/172.18.19.184/tcp/4626/ws/p2p/16Uiu2HAm7rADiYA1qTp963e1m51FsA2nKhnr9TeUFiBr6xbo6gSr"],"enrUri":"enr:-L24QFTsFfIhJVN-6zxgcCJyH6YoveDLcu_kkr1yvowoRPDFWsXCMr536Q0jdArKRgXiLHOx_dAYmspCJl7Gk672YYMCgmlkgnY0gmlwhKwSE7iKbXVsdGlhZGRyc5YACASsEhO4BhIRAAoErBITuAYSEt0DgnJzhQACAQAAiXNlY3AyNTZrMaECuJIdGY6-pYoIM_4-wNb3tAbzINY1hcPmEgQNO0K6Bn-DdGNwghIRg3VkcIISE4V3YWt1MgU"}'
INFO  src.node.waku_node:waku_node.py:292 REST service is ready !!
DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest
DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node2_2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:90 Starting Node...
DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku
DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists
DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.153.76
DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['30743', '30744', '30745', '30746', '30747']
DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2
DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args
DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set
INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN
DEBUG  src.node.waku_node:waku_node.py:178 Using volumes []
DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 30743:30743 -p 30744:30744 -p 30745:30745 -p 30746:30746 -p 30747:30747 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=30745 --rest-port=30743 --tcp-port=30744 --discv5-udp-port=30746 --rest-address=0.0.0.0 --nat=extip:172.18.153.76 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=b1f3d3a8cdaa6db010dcfdf54849adeafea854aeff6c3111ade8ae3aff9197c9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=30747 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFTsFfIhJVN-6zxgcCJyH6YoveDLcu_kkr1yvowoRPDFWsXCMr536Q0jdArKRgXiLHOx_dAYmspCJl7Gk672YYMCgmlkgnY0gmlwhKwSE7iKbXVsdGlhZGRyc5YACASsEhO4BhIRAAoErBITuAYSEt0DgnJzhQACAQAAiXNlY3AyNTZrMaECuJIdGY6-pYoIM_4-wNb3tAbzINY1hcPmEgQNO0K6Bn-DdGNwghIRg3VkcIISE4V3YWt1MgU
DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.153.76 waku 539bb3f626e959b61a353585747bd7ad31bd34b6214a823fc7d9e94cdff67e17
DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID 539bb3f626e9. Setting up logs at ./log/docker/node2_2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 30743
DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:30743/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:30743/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:30743/debug/v1/info" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.153.76/tcp/30744/p2p/16Uiu2HAm9ghMLHAM8tikiS1qPrmAecikG9Vz2jSRNZFCfHdE991J","/ip4/172.18.153.76/tcp/30745/ws/p2p/16Uiu2HAm9ghMLHAM8tikiS1qPrmAecikG9Vz2jSRNZFCfHdE991J"],"enrUri":"enr:-L24QIPuYX7Q8ocudccQ5zNadiAhEcloykd3ZDQy4yRixDMCNqIbG-F9uwY7ovz8ap9Wc1Sm_k-RyVDtRV8C2CmkpqECgmlkgnY0gmlwhKwSmUyKbXVsdGlhZGRyc5YACASsEplMBngYAAoErBKZTAZ4Gd0DgnJzhQACAQAAiXNlY3AyNTZrMaEC09zhNX8SL4OTt1zzcmNeod_zdm_cvpnpEw00yQvmwMGDdGNwgngYg3VkcIJ4GoV3YWt1MgE"}'
INFO  src.node.waku_node:waku_node.py:292 REST service is ready !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:30743/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.19.184/tcp/4625/p2p/16Uiu2HAm7rADiYA1qTp963e1m51FsA2nKhnr9TeUFiBr6xbo6gSr"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:4624/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7", "/waku/2/rs/2/8", "/waku/2/rs/2/9", "/waku/2/rs/2/10", "/waku/2/rs/2/11", "/waku/2/rs/2/12", "/waku/2/rs/2/13", "/waku/2/rs/2/14", "/waku/2/rs/2/15", "/waku/2/rs/2/16", "/waku/2/rs/2/17", "/waku/2/rs/2/18", "/waku/2/rs/2/19", "/waku/2/rs/2/20", "/waku/2/rs/2/21", "/waku/2/rs/2/22", "/waku/2/rs/2/23", "/waku/2/rs/2/24", "/waku/2/rs/2/25", "/waku/2/rs/2/26", "/waku/2/rs/2/27", "/waku/2/rs/2/28", "/waku/2/rs/2/29", "/waku/2/rs/2/30", "/waku/2/rs/2/31", "/waku/2/rs/2/32", "/waku/2/rs/2/33", "/waku/2/rs/2/34", "/waku/2/rs/2/35", "/waku/2/rs/2/36", "/waku/2/rs/2/37", "/waku/2/rs/2/38", "/waku/2/rs/2/39", "/waku/2/rs/2/40", "/waku/2/rs/2/41", "/waku/2/rs/2/42", "/waku/2/rs/2/43", "/waku/2/rs/2/44", "/waku/2/rs/2/45", "/waku/2/rs/2/46", "/waku/2/rs/2/47", "/waku/2/rs/2/48", "/waku/2/rs/2/49"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:30743/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7", "/waku/2/rs/2/8", "/waku/2/rs/2/9", "/waku/2/rs/2/10", "/waku/2/rs/2/11", "/waku/2/rs/2/12", "/waku/2/rs/2/13", "/waku/2/rs/2/14", "/waku/2/rs/2/15", "/waku/2/rs/2/16", "/waku/2/rs/2/17", "/waku/2/rs/2/18", "/waku/2/rs/2/19", "/waku/2/rs/2/20", "/waku/2/rs/2/21", "/waku/2/rs/2/22", "/waku/2/rs/2/23", "/waku/2/rs/2/24", "/waku/2/rs/2/25", "/waku/2/rs/2/26", "/waku/2/rs/2/27", "/waku/2/rs/2/28", "/waku/2/rs/2/29", "/waku/2/rs/2/30", "/waku/2/rs/2/31", "/waku/2/rs/2/32", "/waku/2/rs/2/33", "/waku/2/rs/2/34", "/waku/2/rs/2/35", "/waku/2/rs/2/36", "/waku/2/rs/2/37", "/waku/2/rs/2/38", "/waku/2/rs/2/39", "/waku/2/rs/2/40", "/waku/2/rs/2/41", "/waku/2/rs/2/42", "/waku/2/rs/2/43", "/waku/2/rs/2/44", "/waku/2/rs/2/45", "/waku/2/rs/2/46", "/waku/2/rs/2/47", "/waku/2/rs/2/48", "/waku/2/rs/2/49"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:4624/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
DEBUG  src.libs.common:common.py:47 Sleeping for 0.1 seconds
DEBUG  src.steps.sharding:sharding.py:158 Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:4624/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685266122096812,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685267466337603,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685268126311636,"ephemeral":false,"proof":""}]'
DEBUG  tests.conftest:conftest.py:59 Running fixture teardown: test_setup
DEBUG  tests.conftest:conftest.py:83 Running fixture teardown: close_open_nodes
DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id 26fedc537f6c
DEBUG  src.node.waku_node:waku_node.py:241 Container stopped.
DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id 539bb3f626e9
DEBUG  src.node.waku_node:waku_node.py:241 Container stopped.
DEBUG  tests.conftest:conftest.py:98 Running fixture teardown: check_waku_log_errors
DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.
DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.
DEBUG  tests.conftest:conftest.py:73 Running fixture teardown: attach_logs_on_fail
DEBUG  tests.conftest:conftest.py:74 Test failed, attempting to attach logs to the allure reports
DEBUG  src.libs.common:common.py:42 Attaching file ./log/docker/node2_2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0__wakuorg_nwaku:latest.log
DEBUG  src.libs.common:common.py:42 Attaching file ./log/docker/node1_2026-03-05_04-34-24__167a9e1e-8c9e-45c2-98ed-f962467fb1a0__wakuorg_nwaku:latest.log
DEBUG  tests.conftest:conftest.py:51 Running fixture setup: test_id
DEBUG  tests.conftest:conftest.py:57 Running test: test_start_node_with_50_pubsub_topics with id: 2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912
DEBUG  src.steps.common:common.py:19 Running fixture setup: common_setup
DEBUG  src.steps.relay:relay.py:28 Running fixture setup: relay_setup
DEBUG  src.steps.sharding:sharding.py:30 Running fixture setup: sharding_setup
DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest
DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node1_2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:90 Starting Node...
DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku
DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists
DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.39.87
DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['54643', '54644', '54645', '54646', '54647']
DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2
DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args
DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set
INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN
DEBUG  src.node.waku_node:waku_node.py:178 Using volumes []
DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 54643:54643 -p 54644:54644 -p 54645:54645 -p 54646:54646 -p 54647:54647 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=54645 --rest-port=54643 --tcp-port=54644 --discv5-udp-port=54646 --rest-address=0.0.0.0 --nat=extip:172.18.39.87 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=e2b701cbefdadc40a8a279f95cfcd6cb2f7d6f30cf46884df5e86aedacb6a8ec --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=54647 --metrics-logging=true --relay=true --filter=true
DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.39.87 waku 88fa90a72427cae659c78735cc9750e6ab256382b8bbf3b3fec31d00ef26ec63
DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID 88fa90a72427. Setting up logs at ./log/docker/node1_2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 54643
DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds
ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container 26fedc537f6c. Exiting log stream.
ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container 539bb3f626e9. Exiting log stream.
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:54643/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:54643/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:54643/debug/v1/info" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.39.87/tcp/54644/p2p/16Uiu2HAmNjZ7fS9zhKqMXJBLSMtjn3VT2XKoVmmDfWWJxCeCTSqC","/ip4/172.18.39.87/tcp/54645/ws/p2p/16Uiu2HAmNjZ7fS9zhKqMXJBLSMtjn3VT2XKoVmmDfWWJxCeCTSqC"],"enrUri":"enr:-L24QATLqZRln1yrVyR4Glk80QZRAm-Sf4YvcQ_gJiG49GNmKmLSiS8sfcR9cVRGOKfi3eFSiYz-zTYg-vUklQ0cHJcCgmlkgnY0gmlwhKwSJ1eKbXVsdGlhZGRyc5YACASsEidXBtV0AAoErBInVwbVdd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDlcA9dXPJWH5lv1I7aqMwzIV-M7qlCpMC5tes5EUkeI-DdGNwgtV0g3VkcILVdoV3YWt1MgU"}'
INFO  src.node.waku_node:waku_node.py:292 REST service is ready !!
DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest
DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node2_2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:90 Starting Node...
DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku
DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists
DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.14.152
DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['19559', '19560', '19561', '19562', '19563']
DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2
DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args
DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set
INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN
DEBUG  src.node.waku_node:waku_node.py:178 Using volumes []
DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 19559:19559 -p 19560:19560 -p 19561:19561 -p 19562:19562 -p 19563:19563 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=19561 --rest-port=19559 --tcp-port=19560 --discv5-udp-port=19562 --rest-address=0.0.0.0 --nat=extip:172.18.14.152 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=fdf002a0ecd5ebe96fdaaaced6bcee57554b6e1448a85fa1c6220d65dda0acae --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=19563 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QATLqZRln1yrVyR4Glk80QZRAm-Sf4YvcQ_gJiG49GNmKmLSiS8sfcR9cVRGOKfi3eFSiYz-zTYg-vUklQ0cHJcCgmlkgnY0gmlwhKwSJ1eKbXVsdGlhZGRyc5YACASsEidXBtV0AAoErBInVwbVdd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDlcA9dXPJWH5lv1I7aqMwzIV-M7qlCpMC5tes5EUkeI-DdGNwgtV0g3VkcILVdoV3YWt1MgU
DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.14.152 waku 39f4095cf877f257866cb7bd17aa6fe8de23e022128117ae6dd295888f5746aa
DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID 39f4095cf877. Setting up logs at ./log/docker/node2_2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 19559
DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:19559/debug/v1/info" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.14.152/tcp/19560/p2p/16Uiu2HAkzNGXMSaK8NvnLw9tGr9FVUbq1bdrnF5AXtGTCu5Hav7B","/ip4/172.18.14.152/tcp/19561/ws/p2p/16Uiu2HAkzNGXMSaK8NvnLw9tGr9FVUbq1bdrnF5AXtGTCu5Hav7B"],"enrUri":"enr:-L24QPZVuLWAZzfVCzBw8Fs16SXwRAMy0G2mpZJmSPOIt0c5aEVbASpLjnpUsBxN8UyRKESIcVj0W_Vr6WrPuYJK7OQCgmlkgnY0gmlwhKwSDpiKbXVsdGlhZGRyc5YACASsEg6YBkxoAAoErBIOmAZMad0DgnJzhQACAQAAiXNlY3AyNTZrMaECSWsrqpJVEna0WrOkdUhcN78fhdxTA55iWLLYRJcSpQKDdGNwgkxog3VkcIJMaoV3YWt1MgE"}'
INFO  src.node.waku_node:waku_node.py:292 REST service is ready !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:19559/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.39.87/tcp/54644/p2p/16Uiu2HAmNjZ7fS9zhKqMXJBLSMtjn3VT2XKoVmmDfWWJxCeCTSqC"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:54643/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7", "/waku/2/rs/2/8", "/waku/2/rs/2/9", "/waku/2/rs/2/10", "/waku/2/rs/2/11", "/waku/2/rs/2/12", "/waku/2/rs/2/13", "/waku/2/rs/2/14", "/waku/2/rs/2/15", "/waku/2/rs/2/16", "/waku/2/rs/2/17", "/waku/2/rs/2/18", "/waku/2/rs/2/19", "/waku/2/rs/2/20", "/waku/2/rs/2/21", "/waku/2/rs/2/22", "/waku/2/rs/2/23", "/waku/2/rs/2/24", "/waku/2/rs/2/25", "/waku/2/rs/2/26", "/waku/2/rs/2/27", "/waku/2/rs/2/28", "/waku/2/rs/2/29", "/waku/2/rs/2/30", "/waku/2/rs/2/31", "/waku/2/rs/2/32", "/waku/2/rs/2/33", "/waku/2/rs/2/34", "/waku/2/rs/2/35", "/waku/2/rs/2/36", "/waku/2/rs/2/37", "/waku/2/rs/2/38", "/waku/2/rs/2/39", "/waku/2/rs/2/40", "/waku/2/rs/2/41", "/waku/2/rs/2/42", "/waku/2/rs/2/43", "/waku/2/rs/2/44", "/waku/2/rs/2/45", "/waku/2/rs/2/46", "/waku/2/rs/2/47", "/waku/2/rs/2/48", "/waku/2/rs/2/49"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:19559/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7", "/waku/2/rs/2/8", "/waku/2/rs/2/9", "/waku/2/rs/2/10", "/waku/2/rs/2/11", "/waku/2/rs/2/12", "/waku/2/rs/2/13", "/waku/2/rs/2/14", "/waku/2/rs/2/15", "/waku/2/rs/2/16", "/waku/2/rs/2/17", "/waku/2/rs/2/18", "/waku/2/rs/2/19", "/waku/2/rs/2/20", "/waku/2/rs/2/21", "/waku/2/rs/2/22", "/waku/2/rs/2/23", "/waku/2/rs/2/24", "/waku/2/rs/2/25", "/waku/2/rs/2/26", "/waku/2/rs/2/27", "/waku/2/rs/2/28", "/waku/2/rs/2/29", "/waku/2/rs/2/30", "/waku/2/rs/2/31", "/waku/2/rs/2/32", "/waku/2/rs/2/33", "/waku/2/rs/2/34", "/waku/2/rs/2/35", "/waku/2/rs/2/36", "/waku/2/rs/2/37", "/waku/2/rs/2/38", "/waku/2/rs/2/39", "/waku/2/rs/2/40", "/waku/2/rs/2/41", "/waku/2/rs/2/42", "/waku/2/rs/2/43", "/waku/2/rs/2/44", "/waku/2/rs/2/45", "/waku/2/rs/2/46", "/waku/2/rs/2/47", "/waku/2/rs/2/48", "/waku/2/rs/2/49"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:54643/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
DEBUG  src.libs.common:common.py:47 Sleeping for 0.1 seconds
DEBUG  src.steps.sharding:sharding.py:158 Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:54643/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685271337804012,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685268126311636,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685273771170899,"ephemeral":false,"proof":""}]'
DEBUG  tests.conftest:conftest.py:59 Running fixture teardown: test_setup
DEBUG  tests.conftest:conftest.py:83 Running fixture teardown: close_open_nodes
DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id 88fa90a72427
DEBUG  src.node.waku_node:waku_node.py:241 Container stopped.
DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id 39f4095cf877
DEBUG  src.node.waku_node:waku_node.py:241 Container stopped.
DEBUG  tests.conftest:conftest.py:98 Running fixture teardown: check_waku_log_errors
DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.
DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.
DEBUG  tests.conftest:conftest.py:73 Running fixture teardown: attach_logs_on_fail
DEBUG  tests.conftest:conftest.py:74 Test failed, attempting to attach logs to the allure reports
DEBUG  src.libs.common:common.py:42 Attaching file ./log/docker/node1_2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912__wakuorg_nwaku:latest.log
DEBUG  src.libs.common:common.py:42 Attaching file ./log/docker/node2_2026-03-05_04-34-29__a390eb93-d425-419f-8a4a-b2a8184c2912__wakuorg_nwaku:latest.log
DEBUG  tests.conftest:conftest.py:51 Running fixture setup: test_id
DEBUG  tests.conftest:conftest.py:57 Running test: test_start_node_with_50_pubsub_topics with id: 2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77
DEBUG  src.steps.common:common.py:19 Running fixture setup: common_setup
DEBUG  src.steps.relay:relay.py:28 Running fixture setup: relay_setup
DEBUG  src.steps.sharding:sharding.py:30 Running fixture setup: sharding_setup
DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest
DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node1_2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:90 Starting Node...
DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku
DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists
DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.209.217
DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['46223', '46224', '46225', '46226', '46227']
DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2
DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args
DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set
INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN
DEBUG  src.node.waku_node:waku_node.py:178 Using volumes []
DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 46223:46223 -p 46224:46224 -p 46225:46225 -p 46226:46226 -p 46227:46227 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=46225 --rest-port=46223 --tcp-port=46224 --discv5-udp-port=46226 --rest-address=0.0.0.0 --nat=extip:172.18.209.217 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=4c7bda39c7baae4afe96a8cfa3a5dcb2ba71ca42a4da64bfc459760c81b1bae0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=46227 --metrics-logging=true --relay=true --filter=true
DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.209.217 waku 05d016f750f930ef08d7a433274d113ce0417a472a8559fa5877a2aacc95f64c
DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID 05d016f750f9. Setting up logs at ./log/docker/node1_2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 46223
DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds
ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container 88fa90a72427. Exiting log stream.
ERROR  src.node.docker_mananger:docker_mananger.py:89 Max retries reached for container 39f4095cf877. Exiting log stream.
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/debug/v1/info" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.209.217/tcp/46224/p2p/16Uiu2HAmBi4WvDyxD4FF55QYpU2BBX88JAn9sU8LEWvuV2edq6pc","/ip4/172.18.209.217/tcp/46225/ws/p2p/16Uiu2HAmBi4WvDyxD4FF55QYpU2BBX88JAn9sU8LEWvuV2edq6pc"],"enrUri":"enr:-L24QHnA0_gn1wfVtgauEC8aQQBjBUbLZN6fY4YNy9s9Ua-4a8prfrLGc_r501Kk0Qy0-vJdmzVl2I-ffL1NRsYclfYCgmlkgnY0gmlwhKwS0dmKbXVsdGlhZGRyc5YACASsEtHZBrSQAAoErBLR2Qa0kd0DgnJzhQACAQAAiXNlY3AyNTZrMaEC8e3EGiAYaQiotoOIvq0-PJQLGEQ7ukSkBO2TlnzI-J2DdGNwgrSQg3VkcIK0koV3YWt1MgU"}'
INFO  src.node.waku_node:waku_node.py:292 REST service is ready !!
DEBUG  src.node.docker_mananger:docker_mananger.py:19 Docker client initialized with image wakuorg/nwaku:latest
DEBUG  src.node.waku_node:waku_node.py:86 WakuNode instance initialized with log path ./log/docker/node2_2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:90 Starting Node...
DEBUG  src.node.docker_mananger:docker_mananger.py:22 Attempting to create or retrieve network waku
DEBUG  src.node.docker_mananger:docker_mananger.py:25 Network waku already exists
DEBUG  src.node.docker_mananger:docker_mananger.py:108 Generated random external IP 172.18.62.143
DEBUG  src.node.docker_mananger:docker_mananger.py:101 Generated ports ['17326', '17327', '17328', '17329', '17330']
DEBUG  src.node.waku_node:waku_node.py:72 Cluster id was resolved to: 2
DEBUG  src.node.waku_node:waku_node.py:163 Removing pubsub-topic from nwaku args
DEBUG  src.node.waku_node:waku_node.py:439 RLN credentials were not set
INFO  src.node.waku_node:waku_node.py:176 RLN credentials not set or credential store not available, starting without RLN
DEBUG  src.node.waku_node:waku_node.py:178 Using volumes []
DEBUG  src.node.docker_mananger:docker_mananger.py:49 docker run -i -t -p 17326:17326 -p 17327:17327 -p 17328:17328 -p 17329:17329 -p 17330:17330 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17328 --rest-port=17326 --tcp-port=17327 --discv5-udp-port=17329 --rest-address=0.0.0.0 --nat=extip:172.18.62.143 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=0a4e2acbf46ffad6a68df8bcf095d8a94b918a09ff10cdac2aaf599e6188e29d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17330 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHnA0_gn1wfVtgauEC8aQQBjBUbLZN6fY4YNy9s9Ua-4a8prfrLGc_r501Kk0Qy0-vJdmzVl2I-ffL1NRsYclfYCgmlkgnY0gmlwhKwS0dmKbXVsdGlhZGRyc5YACASsEtHZBrSQAAoErBLR2Qa0kd0DgnJzhQACAQAAiXNlY3AyNTZrMaEC8e3EGiAYaQiotoOIvq0-PJQLGEQ7ukSkBO2TlnzI-J2DdGNwgrSQg3VkcIK0koV3YWt1MgU
DEBUG  src.node.docker_mananger:docker_mananger.py:55 docker network connect --ip 172.18.62.143 waku ca160042858a92e162c5b461a77df24bd4a4bd950d452c207e20ae98798b1b27
DEBUG  src.node.docker_mananger:docker_mananger.py:58 Container started with ID ca160042858a. Setting up logs at ./log/docker/node2_2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77__wakuorg_nwaku:latest.log
DEBUG  src.node.waku_node:waku_node.py:190 Started container from image wakuorg/nwaku:latest. REST: 17326
DEBUG  src.libs.common:common.py:47 Sleeping for 1 seconds
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:17326/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:17326/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:17326/health" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
INFO  src.node.waku_node:waku_node.py:287 Node protocols are initialized !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:17326/debug/v1/info" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.62.143/tcp/17327/p2p/16Uiu2HAmEb55MgeNaLx58b6Yd7VDmZxg6SXH8vq8iYTGBJ7feELn","/ip4/172.18.62.143/tcp/17328/ws/p2p/16Uiu2HAmEb55MgeNaLx58b6Yd7VDmZxg6SXH8vq8iYTGBJ7feELn"],"enrUri":"enr:-L24QDaEUu_L4moXSxIGAjMbcsatvjy6Iblez_2lFLkskuB2A7t4g9tlcNyO410ZQ4dhZgTbXuMRPJlFpg4uAOoBSLUCgmlkgnY0gmlwhKwSPo-KbXVsdGlhZGRyc5YACASsEj6PBkOvAAoErBI-jwZDsN0DgnJzhQACAQAAiXNlY3AyNTZrMaEDHLZkvpZILsadwPcbA-40tFbck0jMzhXPvu_2J8dFQ3eDdGNwgkOvg3VkcIJDsYV3YWt1MgE"}'
INFO  src.node.waku_node:waku_node.py:292 REST service is ready !!
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:17326/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.209.217/tcp/46224/p2p/16Uiu2HAmBi4WvDyxD4FF55QYpU2BBX88JAn9sU8LEWvuV2edq6pc"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:46223/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7", "/waku/2/rs/2/8", "/waku/2/rs/2/9", "/waku/2/rs/2/10", "/waku/2/rs/2/11", "/waku/2/rs/2/12", "/waku/2/rs/2/13", "/waku/2/rs/2/14", "/waku/2/rs/2/15", "/waku/2/rs/2/16", "/waku/2/rs/2/17", "/waku/2/rs/2/18", "/waku/2/rs/2/19", "/waku/2/rs/2/20", "/waku/2/rs/2/21", "/waku/2/rs/2/22", "/waku/2/rs/2/23", "/waku/2/rs/2/24", "/waku/2/rs/2/25", "/waku/2/rs/2/26", "/waku/2/rs/2/27", "/waku/2/rs/2/28", "/waku/2/rs/2/29", "/waku/2/rs/2/30", "/waku/2/rs/2/31", "/waku/2/rs/2/32", "/waku/2/rs/2/33", "/waku/2/rs/2/34", "/waku/2/rs/2/35", "/waku/2/rs/2/36", "/waku/2/rs/2/37", "/waku/2/rs/2/38", "/waku/2/rs/2/39", "/waku/2/rs/2/40", "/waku/2/rs/2/41", "/waku/2/rs/2/42", "/waku/2/rs/2/43", "/waku/2/rs/2/44", "/waku/2/rs/2/45", "/waku/2/rs/2/46", "/waku/2/rs/2/47", "/waku/2/rs/2/48", "/waku/2/rs/2/49"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:17326/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0", "/waku/2/rs/2/1", "/waku/2/rs/2/2", "/waku/2/rs/2/3", "/waku/2/rs/2/4", "/waku/2/rs/2/5", "/waku/2/rs/2/6", "/waku/2/rs/2/7", "/waku/2/rs/2/8", "/waku/2/rs/2/9", "/waku/2/rs/2/10", "/waku/2/rs/2/11", "/waku/2/rs/2/12", "/waku/2/rs/2/13", "/waku/2/rs/2/14", "/waku/2/rs/2/15", "/waku/2/rs/2/16", "/waku/2/rs/2/17", "/waku/2/rs/2/18", "/waku/2/rs/2/19", "/waku/2/rs/2/20", "/waku/2/rs/2/21", "/waku/2/rs/2/22", "/waku/2/rs/2/23", "/waku/2/rs/2/24", "/waku/2/rs/2/25", "/waku/2/rs/2/26", "/waku/2/rs/2/27", "/waku/2/rs/2/28", "/waku/2/rs/2/29", "/waku/2/rs/2/30", "/waku/2/rs/2/31", "/waku/2/rs/2/32", "/waku/2/rs/2/33", "/waku/2/rs/2/34", "/waku/2/rs/2/35", "/waku/2/rs/2/36", "/waku/2/rs/2/37", "/waku/2/rs/2/38", "/waku/2/rs/2/39", "/waku/2/rs/2/40", "/waku/2/rs/2/41", "/waku/2/rs/2/42", "/waku/2/rs/2/43", "/waku/2/rs/2/44", "/waku/2/rs/2/45", "/waku/2/rs/2/46", "/waku/2/rs/2/47", "/waku/2/rs/2/48", "/waku/2/rs/2/49"]'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X POST "http://127.0.0.1:46223/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'OK'
DEBUG  src.libs.common:common.py:47 Sleeping for 0.1 seconds
DEBUG  src.steps.sharding:sharding.py:158 Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
INFO  src.node.api_clients.base_client:base_client.py:37 curl -v -X GET "http://127.0.0.1:46223/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
INFO  src.node.api_clients.base_client:base_client.py:22 Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685275240035270,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685278853493844,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772685278910100495,"ephemeral":false,"proof":""}]'
DEBUG  tests.conftest:conftest.py:59 Running fixture teardown: test_setup
DEBUG  tests.conftest:conftest.py:83 Running fixture teardown: close_open_nodes
DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id 05d016f750f9
DEBUG  src.node.waku_node:waku_node.py:241 Container stopped.
DEBUG  src.node.waku_node:waku_node.py:234 Stopping container with id ca160042858a
DEBUG  src.node.waku_node:waku_node.py:241 Container stopped.
DEBUG  tests.conftest:conftest.py:98 Running fixture teardown: check_waku_log_errors
DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.
DEBUG  src.node.docker_mananger:docker_mananger.py:144 No errors found in the waku logs.
DEBUG  tests.conftest:conftest.py:73 Running fixture teardown: attach_logs_on_fail
DEBUG  tests.conftest:conftest.py:74 Test failed, attempting to attach logs to the allure reports
DEBUG  src.libs.common:common.py:42 Attaching file ./log/docker/node1_2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77__wakuorg_nwaku:latest.log
DEBUG  src.libs.common:common.py:42 Attaching file ./log/docker/node2_2026-03-05_04-34-35__a79d0a36-c6af-4428-adea-d04fcc3d0d77__wakuorg_nwaku:latest.log