201 lines
34 KiB
Plaintext

2026-03-06 04:33:58.917 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-06 04:33:58.918 DEBUG [tests.conftest] Running test: test_content_topic_also_in_docker_flags[/toychat/2/huilong/proto] with id: 2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd
2026-03-06 04:33:58.918 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-06 04:33:58.918 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-06 04:33:58.919 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-06 04:33:58.927 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-06 04:33:58.927 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd__wakuorg_nwaku:latest.log
2026-03-06 04:33:58.927 DEBUG [src.node.waku_node] Starting Node...
2026-03-06 04:33:58.927 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-06 04:33:58.929 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-06 04:33:58.929 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.13.228
2026-03-06 04:33:58.929 DEBUG [src.node.docker_mananger] Generated ports ['65292', '65293', '65294', '65295', '65296']
2026-03-06 04:33:58.929 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-06 04:33:58.929 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-06 04:33:58.930 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-06 04:33:58.930 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-06 04:33:58.930 DEBUG [src.node.waku_node] Using volumes []
2026-03-06 04:33:58.930 DEBUG [src.node.docker_mananger] docker run -i -t -p 65292:65292 -p 65293:65293 -p 65294:65294 -p 65295:65295 -p 65296:65296 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=65294 --rest-port=65292 --tcp-port=65293 --discv5-udp-port=65295 --rest-address=0.0.0.0 --nat=extip:172.18.13.228 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f34fc2f9e7a7b69d6deea6658f01ae6a91ae9dba9b9bfcaeef753e2aa98ef08f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=65296 --metrics-logging=true --relay=true --filter=true --content-topic=/toychat/2/huilong/proto
2026-03-06 04:33:59.127 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.13.228 waku 1e7d965f0b267e820007ccbeae5ec7211638aceee4f7a630ded2ba8e7acc59e0
2026-03-06 04:33:59.154 ERROR [src.node.docker_mananger] Max retries reached for container 9d2eaa75ae27. Exiting log stream.
2026-03-06 04:33:59.164 DEBUG [src.node.docker_mananger] Container started with ID 1e7d965f0b26. Setting up logs at ./log/docker/node1_2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd__wakuorg_nwaku:latest.log
2026-03-06 04:33:59.165 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 65292
2026-03-06 04:33:59.165 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-06 04:33:59.728 ERROR [src.node.docker_mananger] Max retries reached for container a1c3805ae830. Exiting log stream.
2026-03-06 04:34:00.166 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65292/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:00.170 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:00.270 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65292/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:00.273 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:00.373 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65292/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:00.376 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:00.477 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65292/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:00.480 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-06 04:34:00.480 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-06 04:34:00.480 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65292/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:00.483 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.13.228/tcp/65293/p2p/16Uiu2HAmL3nhWrrqoUZEEV6vpJk9dtYJxu81vD4jZZbUSPXiJjD6","/ip4/172.18.13.228/tcp/65294/ws/p2p/16Uiu2HAmL3nhWrrqoUZEEV6vpJk9dtYJxu81vD4jZZbUSPXiJjD6"],"enrUri":"enr:-L24QM0Nj7_wSf11AGQy6NxPV602bMi7mWd2opPmZ187ogsaVogdMOunzWUT4irsjBB-jNXJUJL_BU5rV0X994QnSVgCgmlkgnY0gmlwhKwSDeSKbXVsdGlhZGRyc5YACASsEg3kBv8NAAoErBIN5Ab_Dt0DgnJzhQACAQAAiXNlY3AyNTZrMaEDbdj-Mea01mBLBSNbydmGOGwBcrIzb8DQN8D_fDPwZp2DdGNwgv8Ng3VkcIL_D4V3YWt1MgU"}'
2026-03-06 04:34:00.483 INFO [src.node.waku_node] REST service is ready !!
2026-03-06 04:34:00.490 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-06 04:34:00.490 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd__wakuorg_nwaku:latest.log
2026-03-06 04:34:00.491 DEBUG [src.node.waku_node] Starting Node...
2026-03-06 04:34:00.491 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-06 04:34:00.492 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-06 04:34:00.492 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.107.184
2026-03-06 04:34:00.493 DEBUG [src.node.docker_mananger] Generated ports ['17170', '17171', '17172', '17173', '17174']
2026-03-06 04:34:00.493 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-06 04:34:00.493 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-06 04:34:00.493 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-06 04:34:00.493 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-06 04:34:00.493 DEBUG [src.node.waku_node] Using volumes []
2026-03-06 04:34:00.493 DEBUG [src.node.docker_mananger] docker run -i -t -p 17170:17170 -p 17171:17171 -p 17172:17172 -p 17173:17173 -p 17174:17174 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17172 --rest-port=17170 --tcp-port=17171 --discv5-udp-port=17173 --rest-address=0.0.0.0 --nat=extip:172.18.107.184 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=e0b865bae4771d0ff1d9cea7dc6fcfaa1c0dd8ddafbedc6f6c3e750ae80b4f3f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17174 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QM0Nj7_wSf11AGQy6NxPV602bMi7mWd2opPmZ187ogsaVogdMOunzWUT4irsjBB-jNXJUJL_BU5rV0X994QnSVgCgmlkgnY0gmlwhKwSDeSKbXVsdGlhZGRyc5YACASsEg3kBv8NAAoErBIN5Ab_Dt0DgnJzhQACAQAAiXNlY3AyNTZrMaEDbdj-Mea01mBLBSNbydmGOGwBcrIzb8DQN8D_fDPwZp2DdGNwgv8Ng3VkcIL_D4V3YWt1MgU --content-topic=/toychat/2/huilong/proto
2026-03-06 04:34:00.701 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.107.184 waku e581867325e63f5e2cd46c9350566d997dd11f112de8cd317fdd336ece795749
2026-03-06 04:34:00.739 DEBUG [src.node.docker_mananger] Container started with ID e581867325e6. Setting up logs at ./log/docker/node2_2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd__wakuorg_nwaku:latest.log
2026-03-06 04:34:00.740 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 17170
2026-03-06 04:34:00.740 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-06 04:34:01.741 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:01.745 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:01.845 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:01.848 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:01.949 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:01.952 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.052 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.055 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.156 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.158 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.259 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.262 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.362 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.365 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.465 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.468 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.569 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.572 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.672 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.675 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.775 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.778 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.879 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:02.882 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:02.982 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:03.012 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:03.113 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:03.116 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-06 04:34:03.116 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-06 04:34:03.117 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17170/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:03.119 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.107.184/tcp/17171/p2p/16Uiu2HAm98o46m3wRzfMGTkZj1LBpEqqwWfmxoR4rZ8AgYwuMuxD","/ip4/172.18.107.184/tcp/17172/ws/p2p/16Uiu2HAm98o46m3wRzfMGTkZj1LBpEqqwWfmxoR4rZ8AgYwuMuxD"],"enrUri":"enr:-L24QDVXG5vtWsfjZqvSgpcqjwRJLV7n8IGch52LACYakKTLBcE7kQ9Qg4AmjeM9PobPyzeE7Prdo5rhrA7IyyP_YpYCgmlkgnY0gmlwhKwSa7iKbXVsdGlhZGRyc5YACASsEmu4BkMTAAoErBJruAZDFN0DgnJzhQACAQAAiXNlY3AyNTZrMaECy7C9-gr9dkRb_2ggtHL2sqmVLahU1EzIsHxvt-SGrDKDdGNwgkMTg3VkcIJDFYV3YWt1MgE"}'
2026-03-06 04:34:03.119 INFO [src.node.waku_node] REST service is ready !!
2026-03-06 04:34:03.119 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17170/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.13.228/tcp/65293/p2p/16Uiu2HAmL3nhWrrqoUZEEV6vpJk9dtYJxu81vD4jZZbUSPXiJjD6"]'
2026-03-06 04:34:03.143 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:03.144 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:65292/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-06 04:34:03.161 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:03.164 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17170/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-06 04:34:03.169 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:03.170 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:65292/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-06 04:34:03.179 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:03.180 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-06 04:34:03.281 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-06 04:34:03.281 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65292/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:03.285 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772771638825167744,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772771643170040271,"ephemeral":false,"proof":""}]'
2026-03-06 04:34:03.329 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-06 04:34:03.331 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-06 04:34:03.331 DEBUG [src.node.waku_node] Stopping container with id 1e7d965f0b26
2026-03-06 04:34:03.948 DEBUG [src.node.waku_node] Container stopped.
2026-03-06 04:34:03.948 DEBUG [src.node.waku_node] Stopping container with id e581867325e6
2026-03-06 04:34:04.538 DEBUG [src.node.waku_node] Container stopped.
2026-03-06 04:34:04.539 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-06 04:34:04.627 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-06 04:34:04.654 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-06 04:34:04.655 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-06 04:34:04.655 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-06 04:34:04.656 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd__wakuorg_nwaku:latest.log
2026-03-06 04:34:04.658 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-06_04-33-58__fb3dd150-e720-4bd5-a609-a9a19fb0a3fd__wakuorg_nwaku:latest.log
2026-03-06 04:34:04.662 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-06 04:34:04.663 DEBUG [tests.conftest] Running test: test_content_topic_also_in_docker_flags[/toychat/2/huilong/proto] with id: 2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4
2026-03-06 04:34:04.663 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-06 04:34:04.663 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-06 04:34:04.663 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-06 04:34:04.671 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-06 04:34:04.672 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4__wakuorg_nwaku:latest.log
2026-03-06 04:34:04.672 DEBUG [src.node.waku_node] Starting Node...
2026-03-06 04:34:04.672 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-06 04:34:04.673 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-06 04:34:04.674 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.202.142
2026-03-06 04:34:04.674 DEBUG [src.node.docker_mananger] Generated ports ['39223', '39224', '39225', '39226', '39227']
2026-03-06 04:34:04.674 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-06 04:34:04.674 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-06 04:34:04.674 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-06 04:34:04.675 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-06 04:34:04.675 DEBUG [src.node.waku_node] Using volumes []
2026-03-06 04:34:04.675 DEBUG [src.node.docker_mananger] docker run -i -t -p 39223:39223 -p 39224:39224 -p 39225:39225 -p 39226:39226 -p 39227:39227 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=39225 --rest-port=39223 --tcp-port=39224 --discv5-udp-port=39226 --rest-address=0.0.0.0 --nat=extip:172.18.202.142 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=46aebd0c574560ef11eb696c2a66ed2f9fab9062fbbeefacb99ce6cfc7faeffd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=39227 --metrics-logging=true --relay=true --filter=true --content-topic=/toychat/2/huilong/proto
2026-03-06 04:34:04.871 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.202.142 waku 30012212b037c4bdd1456442f4e73f5ba3dd0afa6380983a97fd424e08fb1f39
2026-03-06 04:34:04.906 DEBUG [src.node.docker_mananger] Container started with ID 30012212b037. Setting up logs at ./log/docker/node1_2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4__wakuorg_nwaku:latest.log
2026-03-06 04:34:04.906 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 39223
2026-03-06 04:34:04.906 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-06 04:34:04.918 ERROR [src.node.docker_mananger] Max retries reached for container 1e7d965f0b26. Exiting log stream.
2026-03-06 04:34:05.350 ERROR [src.node.docker_mananger] Max retries reached for container e581867325e6. Exiting log stream.
2026-03-06 04:34:05.908 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39223/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:05.911 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:06.011 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39223/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:06.014 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:06.114 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39223/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:06.125 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:06.225 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39223/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:06.228 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-06 04:34:06.229 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-06 04:34:06.229 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39223/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:06.231 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.202.142/tcp/39224/p2p/16Uiu2HAmQSsVhTsVx7rdvWQNcKmFXijnvZtkWfxkyAhEfCEA78oQ","/ip4/172.18.202.142/tcp/39225/ws/p2p/16Uiu2HAmQSsVhTsVx7rdvWQNcKmFXijnvZtkWfxkyAhEfCEA78oQ"],"enrUri":"enr:-L24QEgjLTcV2C5UtkYWGTR0oOcpVhJGO4WD3wd7sRJBe69tc2zv9so2v3Yl-JXxBMX4edaX-v16iCwY3f5CVlMghe4CgmlkgnY0gmlwhKwSyo6KbXVsdGlhZGRyc5YACASsEsqOBpk4AAoErBLKjgaZOd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDrzGFvG6oYN7meDCaK8nZ-tQMjWJOiVLrImFYIsH91t-DdGNwgpk4g3VkcIKZOoV3YWt1MgU"}'
2026-03-06 04:34:06.231 INFO [src.node.waku_node] REST service is ready !!
2026-03-06 04:34:06.239 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-06 04:34:06.239 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4__wakuorg_nwaku:latest.log
2026-03-06 04:34:06.239 DEBUG [src.node.waku_node] Starting Node...
2026-03-06 04:34:06.239 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-06 04:34:06.241 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-06 04:34:06.241 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.49.83
2026-03-06 04:34:06.242 DEBUG [src.node.docker_mananger] Generated ports ['25235', '25236', '25237', '25238', '25239']
2026-03-06 04:34:06.242 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-06 04:34:06.242 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-06 04:34:06.242 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-06 04:34:06.242 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-06 04:34:06.242 DEBUG [src.node.waku_node] Using volumes []
2026-03-06 04:34:06.243 DEBUG [src.node.docker_mananger] docker run -i -t -p 25235:25235 -p 25236:25236 -p 25237:25237 -p 25238:25238 -p 25239:25239 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=25237 --rest-port=25235 --tcp-port=25236 --discv5-udp-port=25238 --rest-address=0.0.0.0 --nat=extip:172.18.49.83 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=7583aa8ad15ebb49e174c3afd0bcc8ccaa9faab4651bb437b7848cf15eb4e3ca --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=25239 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QEgjLTcV2C5UtkYWGTR0oOcpVhJGO4WD3wd7sRJBe69tc2zv9so2v3Yl-JXxBMX4edaX-v16iCwY3f5CVlMghe4CgmlkgnY0gmlwhKwSyo6KbXVsdGlhZGRyc5YACASsEsqOBpk4AAoErBLKjgaZOd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDrzGFvG6oYN7meDCaK8nZ-tQMjWJOiVLrImFYIsH91t-DdGNwgpk4g3VkcIKZOoV3YWt1MgU --content-topic=/toychat/2/huilong/proto
2026-03-06 04:34:06.444 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.49.83 waku 008471fdb63872f20402e89f95d8d3b6360411e0d44f3be9aa6afbdf25325ecc
2026-03-06 04:34:06.480 DEBUG [src.node.docker_mananger] Container started with ID 008471fdb638. Setting up logs at ./log/docker/node2_2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4__wakuorg_nwaku:latest.log
2026-03-06 04:34:06.481 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 25235
2026-03-06 04:34:06.481 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-06 04:34:07.481 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25235/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:07.484 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:07.584 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25235/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:07.587 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:07.688 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25235/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:07.691 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-06 04:34:07.792 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25235/health" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:07.795 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-06 04:34:07.795 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-06 04:34:07.795 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25235/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:07.797 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.49.83/tcp/25236/p2p/16Uiu2HAm7EPzjxJnxeXzgoVJpvZ3E3tTmUHEm6Zkk8NP55r2qkyw","/ip4/172.18.49.83/tcp/25237/ws/p2p/16Uiu2HAm7EPzjxJnxeXzgoVJpvZ3E3tTmUHEm6Zkk8NP55r2qkyw"],"enrUri":"enr:-L24QDTpiTP94_gX-jRSHZ5CPaMvUQwfHwFeJKBi6gQvQGpPDjJXqEudnKAU4N4yQKycu9ZfASaG_effRwE1KjhX670CgmlkgnY0gmlwhKwSMVOKbXVsdGlhZGRyc5YACASsEjFTBmKUAAoErBIxUwZild0DgnJzhQACAQAAiXNlY3AyNTZrMaECr2jIDEnTR3OwVhoCARPB17mP-dAfaOZPVGwYX02ZWSKDdGNwgmKUg3VkcIJiloV3YWt1MgE"}'
2026-03-06 04:34:07.798 INFO [src.node.waku_node] REST service is ready !!
2026-03-06 04:34:07.798 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25235/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.202.142/tcp/39224/p2p/16Uiu2HAmQSsVhTsVx7rdvWQNcKmFXijnvZtkWfxkyAhEfCEA78oQ"]'
2026-03-06 04:34:07.827 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:07.830 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39223/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-06 04:34:07.838 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:07.839 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25235/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-06 04:34:07.845 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:07.846 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39223/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-06 04:34:07.858 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-06 04:34:07.859 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-06 04:34:07.960 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-06 04:34:07.961 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39223/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-06 04:34:07.963 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772771644889020827,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772771647846395167,"ephemeral":false,"proof":""}]'
2026-03-06 04:34:07.974 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-06 04:34:07.975 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-06 04:34:07.975 DEBUG [src.node.waku_node] Stopping container with id 30012212b037
2026-03-06 04:34:08.566 DEBUG [src.node.waku_node] Container stopped.
2026-03-06 04:34:08.569 DEBUG [src.node.waku_node] Stopping container with id 008471fdb638
2026-03-06 04:34:09.155 DEBUG [src.node.waku_node] Container stopped.
2026-03-06 04:34:09.157 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-06 04:34:09.221 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-06 04:34:09.250 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-06 04:34:09.251 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-06 04:34:09.251 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-06 04:34:09.251 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4__wakuorg_nwaku:latest.log
2026-03-06 04:34:09.253 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-06_04-34-04__3b4600fc-289d-47d3-ae51-fbf747ee8ea4__wakuorg_nwaku:latest.log