194 lines
32 KiB
Plaintext

2026-03-12 04:39:10.984 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-12 04:39:10.985 DEBUG [tests.conftest] Running test: test_single_pubsub_topic[/waku/2/rs/2/0] with id: 2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a
2026-03-12 04:39:10.986 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-12 04:39:10.986 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-12 04:39:10.987 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-12 04:39:10.999 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:39:11.000 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a__wakuorg_nwaku:latest.log
2026-03-12 04:39:11.000 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:39:11.001 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:39:11.003 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:39:11.004 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.156.172
2026-03-12 04:39:11.004 DEBUG [src.node.docker_mananger] Generated ports ['22147', '22148', '22149', '22150', '22151']
2026-03-12 04:39:11.005 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-12 04:39:11.005 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-12 04:39:11.005 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:39:11.006 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:39:11.006 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:39:11.006 DEBUG [src.node.docker_mananger] docker run -i -t -p 22147:22147 -p 22148:22148 -p 22149:22149 -p 22150:22150 -p 22151:22151 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22149 --rest-port=22147 --tcp-port=22148 --discv5-udp-port=22150 --rest-address=0.0.0.0 --nat=extip:172.18.156.172 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=bd7304ad02ede8b2fd0b8b1260d1e2dd0b654ef3fc5e5e5a7bc43c1d1ff02a9f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22151 --metrics-logging=true --relay=true --filter=true
2026-03-12 04:39:11.182 ERROR [src.node.docker_mananger] Max retries reached for container 48c02d571b61. Exiting log stream.
2026-03-12 04:39:11.217 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.156.172 waku 72985bdfc1570b7a93d249489dec7921b5b62da51be5baa8aa62733abf2dda28
2026-03-12 04:39:11.252 DEBUG [src.node.docker_mananger] Container started with ID 72985bdfc157. Setting up logs at ./log/docker/node1_2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a__wakuorg_nwaku:latest.log
2026-03-12 04:39:11.253 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22147
2026-03-12 04:39:11.253 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:39:11.931 ERROR [src.node.docker_mananger] Max retries reached for container b84e6bc57460. Exiting log stream.
2026-03-12 04:39:12.253 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22147/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:12.256 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:12.356 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22147/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:12.359 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:12.459 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22147/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:12.462 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:39:12.462 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:39:12.462 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22147/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:12.464 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.156.172/tcp/22148/p2p/16Uiu2HAmNrzxh9WNPKVFqVeNi1nVfK29MA4bX8sqLbNeprkngBpc","/ip4/172.18.156.172/tcp/22149/ws/p2p/16Uiu2HAmNrzxh9WNPKVFqVeNi1nVfK29MA4bX8sqLbNeprkngBpc"],"enrUri":"enr:-L24QA0mfY8IcJBKMoIV_BHa0kYKHY7o_IBwaeD4IzpZOqAzEJ8WL_l36G71Y3JO2kx6-erzowriZW6tYNcHfpXC0WsCgmlkgnY0gmlwhKwSnKyKbXVsdGlhZGRyc5YACASsEpysBlaEAAoErBKcrAZWhd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDl6iHdoiN4p9VjnuGWw35Zrc6yNX-fy_qrhNlkGzfvrmDdGNwglaEg3VkcIJWhoV3YWt1MgU"}'
2026-03-12 04:39:12.464 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:39:12.471 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:39:12.471 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a__wakuorg_nwaku:latest.log
2026-03-12 04:39:12.472 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:39:12.472 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:39:12.473 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:39:12.473 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.99.8
2026-03-12 04:39:12.473 DEBUG [src.node.docker_mananger] Generated ports ['24035', '24036', '24037', '24038', '24039']
2026-03-12 04:39:12.474 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-12 04:39:12.474 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-12 04:39:12.474 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:39:12.474 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:39:12.474 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:39:12.474 DEBUG [src.node.docker_mananger] docker run -i -t -p 24035:24035 -p 24036:24036 -p 24037:24037 -p 24038:24038 -p 24039:24039 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=24037 --rest-port=24035 --tcp-port=24036 --discv5-udp-port=24038 --rest-address=0.0.0.0 --nat=extip:172.18.99.8 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=cfcb0fc4f74a9ba1fad75e6fdd74d34dbfddf2c6ccbcea1eff60fa4a8e07fdff --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=24039 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QA0mfY8IcJBKMoIV_BHa0kYKHY7o_IBwaeD4IzpZOqAzEJ8WL_l36G71Y3JO2kx6-erzowriZW6tYNcHfpXC0WsCgmlkgnY0gmlwhKwSnKyKbXVsdGlhZGRyc5YACASsEpysBlaEAAoErBKcrAZWhd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDl6iHdoiN4p9VjnuGWw35Zrc6yNX-fy_qrhNlkGzfvrmDdGNwglaEg3VkcIJWhoV3YWt1MgU
2026-03-12 04:39:12.689 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.99.8 waku 5e7e01ef410491018cebc60f19b9261c8acfccb31912bc848fec1e8d9b66a2eb
2026-03-12 04:39:12.726 DEBUG [src.node.docker_mananger] Container started with ID 5e7e01ef4104. Setting up logs at ./log/docker/node2_2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a__wakuorg_nwaku:latest.log
2026-03-12 04:39:12.726 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 24035
2026-03-12 04:39:12.726 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:39:13.727 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24035/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:13.730 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:13.831 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24035/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:13.833 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:13.933 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24035/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:13.936 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:14.036 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24035/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:14.045 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:14.145 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24035/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:14.148 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:39:14.148 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:39:14.148 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24035/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:14.151 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.99.8/tcp/24036/p2p/16Uiu2HAky524Bnv92Vj6tuKUPY3V7gtDtCnmiV5KCccKDrimKGh1","/ip4/172.18.99.8/tcp/24037/ws/p2p/16Uiu2HAky524Bnv92Vj6tuKUPY3V7gtDtCnmiV5KCccKDrimKGh1"],"enrUri":"enr:-L24QGn_a6M-DbV630OlX74sgPY4rlbW_UEAdfJAY2IRGo2kIl6hzvjIpuZMW2Gt8wUVou_HDJPFhgs5F529usUkUDYCgmlkgnY0gmlwhKwSYwiKbXVsdGlhZGRyc5YACASsEmMIBl3kAAoErBJjCAZd5d0DgnJzhQACAQAAiXNlY3AyNTZrMaECNiRAHILXyL3PAzG2ULsEU4OSVI11jNxClVp1wHC8TpyDdGNwgl3kg3VkcIJd5oV3YWt1MgE"}'
2026-03-12 04:39:14.151 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:39:14.151 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:24035/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.156.172/tcp/22148/p2p/16Uiu2HAmNrzxh9WNPKVFqVeNi1nVfK29MA4bX8sqLbNeprkngBpc"]'
2026-03-12 04:39:14.176 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:14.177 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22147/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-12 04:39:14.190 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:14.192 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:24035/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-12 04:39:14.197 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:14.198 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22147/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-12 04:39:14.211 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:14.212 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-12 04:39:14.312 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-12 04:39:14.312 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22147/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:14.315 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773290354020163471,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773290354198345238,"ephemeral":false,"proof":""}]'
2026-03-12 04:39:14.323 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-12 04:39:14.324 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-12 04:39:14.325 DEBUG [src.node.waku_node] Stopping container with id 72985bdfc157
2026-03-12 04:39:14.950 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:39:14.951 DEBUG [src.node.waku_node] Stopping container with id 5e7e01ef4104
2026-03-12 04:39:15.521 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:39:15.523 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-12 04:39:15.557 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:39:15.585 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:39:15.587 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-12 04:39:15.587 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-12 04:39:15.589 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a__wakuorg_nwaku:latest.log
2026-03-12 04:39:15.590 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-12_04-39-10__cf449805-877d-48d4-85e3-ae80dc81b86a__wakuorg_nwaku:latest.log
2026-03-12 04:39:15.594 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-12 04:39:15.594 DEBUG [tests.conftest] Running test: test_single_pubsub_topic[/waku/2/rs/2/0] with id: 2026-03-12_04-39-15__44eeef8e-b71a-44b6-93a3-32190f8cb675
2026-03-12 04:39:15.595 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-12 04:39:15.595 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-12 04:39:15.595 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-12 04:39:15.603 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:39:15.603 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-12_04-39-15__44eeef8e-b71a-44b6-93a3-32190f8cb675__wakuorg_nwaku:latest.log
2026-03-12 04:39:15.603 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:39:15.603 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:39:15.605 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:39:15.605 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.244.144
2026-03-12 04:39:15.605 DEBUG [src.node.docker_mananger] Generated ports ['59809', '59810', '59811', '59812', '59813']
2026-03-12 04:39:15.605 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-12 04:39:15.605 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-12 04:39:15.605 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:39:15.605 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:39:15.605 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:39:15.606 DEBUG [src.node.docker_mananger] docker run -i -t -p 59809:59809 -p 59810:59810 -p 59811:59811 -p 59812:59812 -p 59813:59813 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=59811 --rest-port=59809 --tcp-port=59810 --discv5-udp-port=59812 --rest-address=0.0.0.0 --nat=extip:172.18.244.144 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=bdbadba9be6713b8145e3d72f8524ff055de5ad9cb5e4dedabbaed701bf64bc3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=59813 --metrics-logging=true --relay=true --filter=true
2026-03-12 04:39:15.791 ERROR [src.node.docker_mananger] Max retries reached for container 72985bdfc157. Exiting log stream.
2026-03-12 04:39:15.808 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.244.144 waku 17cf552cc86099d72ed30123b462ac1a3065da1b6a2114ce3f70becfc425a3f8
2026-03-12 04:39:15.842 DEBUG [src.node.docker_mananger] Container started with ID 17cf552cc860. Setting up logs at ./log/docker/node1_2026-03-12_04-39-15__44eeef8e-b71a-44b6-93a3-32190f8cb675__wakuorg_nwaku:latest.log
2026-03-12 04:39:15.842 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 59809
2026-03-12 04:39:15.843 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:39:16.344 ERROR [src.node.docker_mananger] Max retries reached for container 5e7e01ef4104. Exiting log stream.
2026-03-12 04:39:16.843 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59809/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:16.846 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:16.946 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59809/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:16.953 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:17.053 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59809/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:17.056 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:39:17.056 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:39:17.056 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59809/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:17.058 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.244.144/tcp/59810/p2p/16Uiu2HAm61ATt4iJSveDJtMUvNz53cRC5QGWPTWnNYYNR7fmwbhQ","/ip4/172.18.244.144/tcp/59811/ws/p2p/16Uiu2HAm61ATt4iJSveDJtMUvNz53cRC5QGWPTWnNYYNR7fmwbhQ"],"enrUri":"enr:-L24QC8E2JkdtXZeB5r6wyKvDnWo2Fg5axxedO8KyeemPPTCEHuA69c5x9UPNw7E2Rn7tBYi7q6IgEra4W1yVmbRfQUCgmlkgnY0gmlwhKwS9JCKbXVsdGlhZGRyc5YACASsEvSQBumiAAoErBL0kAbpo90DgnJzhQACAQAAiXNlY3AyNTZrMaECnSk-RF--rQr1uki_jnMyfWm-cPeztlw9vG8WU7aJsV-DdGNwgumig3VkcILppIV3YWt1MgU"}'
2026-03-12 04:39:17.059 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:39:17.066 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-12 04:39:17.066 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-12_04-39-15__44eeef8e-b71a-44b6-93a3-32190f8cb675__wakuorg_nwaku:latest.log
2026-03-12 04:39:17.066 DEBUG [src.node.waku_node] Starting Node...
2026-03-12 04:39:17.066 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-12 04:39:17.067 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-12 04:39:17.068 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.70.89
2026-03-12 04:39:17.068 DEBUG [src.node.docker_mananger] Generated ports ['4534', '4535', '4536', '4537', '4538']
2026-03-12 04:39:17.068 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-12 04:39:17.068 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-12 04:39:17.068 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-12 04:39:17.068 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-12 04:39:17.068 DEBUG [src.node.waku_node] Using volumes []
2026-03-12 04:39:17.069 DEBUG [src.node.docker_mananger] docker run -i -t -p 4534:4534 -p 4535:4535 -p 4536:4536 -p 4537:4537 -p 4538:4538 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4536 --rest-port=4534 --tcp-port=4535 --discv5-udp-port=4537 --rest-address=0.0.0.0 --nat=extip:172.18.70.89 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f3e60ec03a075a8af11bdbee6946d5dbde3ee91dd8c7b6c8f85cbc2aa135d01a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4538 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC8E2JkdtXZeB5r6wyKvDnWo2Fg5axxedO8KyeemPPTCEHuA69c5x9UPNw7E2Rn7tBYi7q6IgEra4W1yVmbRfQUCgmlkgnY0gmlwhKwS9JCKbXVsdGlhZGRyc5YACASsEvSQBumiAAoErBL0kAbpo90DgnJzhQACAQAAiXNlY3AyNTZrMaECnSk-RF--rQr1uki_jnMyfWm-cPeztlw9vG8WU7aJsV-DdGNwgumig3VkcILppIV3YWt1MgU
2026-03-12 04:39:17.273 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.70.89 waku f12dc0ff46773f68b9432f1b640413e373a7cdcb6db08393976eb6dab392c8d9
2026-03-12 04:39:17.310 DEBUG [src.node.docker_mananger] Container started with ID f12dc0ff4677. Setting up logs at ./log/docker/node2_2026-03-12_04-39-15__44eeef8e-b71a-44b6-93a3-32190f8cb675__wakuorg_nwaku:latest.log
2026-03-12 04:39:17.310 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 4534
2026-03-12 04:39:17.310 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-12 04:39:18.312 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.315 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:18.415 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.418 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:18.518 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.521 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:18.622 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.624 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:18.725 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.728 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:18.828 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.831 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:18.931 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:18.934 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:19.034 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.037 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:19.138 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.140 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:19.241 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.244 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:19.344 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.375 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-12 04:39:19.476 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/health" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.479 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-12 04:39:19.479 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-12 04:39:19.479 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.482 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.70.89/tcp/4535/p2p/16Uiu2HAmMdjj3Zaauwi5dLz7f87s2KfA9Eiqr3SwD8XShxbx8rF3","/ip4/172.18.70.89/tcp/4536/ws/p2p/16Uiu2HAmMdjj3Zaauwi5dLz7f87s2KfA9Eiqr3SwD8XShxbx8rF3"],"enrUri":"enr:-L24QGsGONgyph48nJEi96EO9BSiz-C4rVVFww1L9-FCd8Q1Ch7jqQHsBi9AT6BegSNv9N0BYzRs_p4yG0Jj4qimjlsCgmlkgnY0gmlwhKwSRlmKbXVsdGlhZGRyc5YACASsEkZZBhG3AAoErBJGWQYRuN0DgnJzhQACAQAAiXNlY3AyNTZrMaEDhWcQqZI8UwxQJr62X1QM1VRK9NtFVyQVBLJOo_D06JqDdGNwghG3g3VkcIIRuYV3YWt1MgE"}'
2026-03-12 04:39:19.482 INFO [src.node.waku_node] REST service is ready !!
2026-03-12 04:39:19.482 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4534/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.244.144/tcp/59810/p2p/16Uiu2HAm61ATt4iJSveDJtMUvNz53cRC5QGWPTWnNYYNR7fmwbhQ"]'
2026-03-12 04:39:19.504 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:19.506 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59809/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-12 04:39:19.518 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:19.518 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4534/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-12 04:39:19.523 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:19.524 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59809/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-12 04:39:19.535 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-12 04:39:19.535 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-12 04:39:19.637 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-12 04:39:19.638 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59809/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.640 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773290359524102430,"ephemeral":false,"proof":""}]'
2026-03-12 04:39:19.642 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-12 04:39:19.642 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4534/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-12 04:39:19.644 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773290359524102430,"ephemeral":false,"proof":""}]'
2026-03-12 04:39:19.647 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-12 04:39:19.649 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-12 04:39:19.649 DEBUG [src.node.waku_node] Stopping container with id 17cf552cc860
2026-03-12 04:39:20.269 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:39:20.270 DEBUG [src.node.waku_node] Stopping container with id f12dc0ff4677
2026-03-12 04:39:20.851 DEBUG [src.node.waku_node] Container stopped.
2026-03-12 04:39:20.853 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-12 04:39:20.883 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-12 04:39:20.911 DEBUG [src.node.docker_mananger] No errors found in the waku logs.