192 lines
32 KiB
Plaintext

2026-03-14 04:34:01.099 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-14 04:34:01.099 DEBUG [tests.conftest] Running test: test_content_topic_also_in_docker_flags[/toychat/2/huilong/proto] with id: 2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a
2026-03-14 04:34:01.100 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-14 04:34:01.100 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-14 04:34:01.100 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-14 04:34:01.107 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-14 04:34:01.108 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a__wakuorg_nwaku:latest.log
2026-03-14 04:34:01.108 DEBUG [src.node.waku_node] Starting Node...
2026-03-14 04:34:01.108 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-14 04:34:01.109 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-14 04:34:01.109 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.114.85
2026-03-14 04:34:01.110 DEBUG [src.node.docker_mananger] Generated ports ['24282', '24283', '24284', '24285', '24286']
2026-03-14 04:34:01.110 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-14 04:34:01.110 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-14 04:34:01.110 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-14 04:34:01.110 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-14 04:34:01.110 DEBUG [src.node.waku_node] Using volumes []
2026-03-14 04:34:01.110 DEBUG [src.node.docker_mananger] docker run -i -t -p 24282:24282 -p 24283:24283 -p 24284:24284 -p 24285:24285 -p 24286:24286 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=24284 --rest-port=24282 --tcp-port=24283 --discv5-udp-port=24285 --rest-address=0.0.0.0 --nat=extip:172.18.114.85 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=3acdfdf9e0c8eefba4123023b97c2ebd31eb580fccbff3c4c1d90fd4c54d408a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=24286 --metrics-logging=true --relay=true --filter=true --content-topic=/toychat/2/huilong/proto
2026-03-14 04:34:01.303 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.114.85 waku b05322eda0a60fb0c3aae4ebd5259fc445dd59283bf598c52d16e431979822dd
2026-03-14 04:34:01.344 DEBUG [src.node.docker_mananger] Container started with ID b05322eda0a6. Setting up logs at ./log/docker/node1_2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a__wakuorg_nwaku:latest.log
2026-03-14 04:34:01.344 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 24282
2026-03-14 04:34:01.346 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-14 04:34:01.351 ERROR [src.node.docker_mananger] Max retries reached for container fe48dae1f255. Exiting log stream.
2026-03-14 04:34:01.898 ERROR [src.node.docker_mananger] Max retries reached for container 91a756219446. Exiting log stream.
2026-03-14 04:34:02.346 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24282/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:02.349 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:02.449 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24282/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:02.490 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:02.591 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24282/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:02.594 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-14 04:34:02.594 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-14 04:34:02.594 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24282/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:02.597 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.114.85/tcp/24283/p2p/16Uiu2HAmNKBYo3XJg3iBwPQsFr8bjpNc25MKAqHXGbkXqhQnCR9V","/ip4/172.18.114.85/tcp/24284/ws/p2p/16Uiu2HAmNKBYo3XJg3iBwPQsFr8bjpNc25MKAqHXGbkXqhQnCR9V"],"enrUri":"enr:-L24QID5RdCxiMY3RrQwjv0W7x_0IDjUcsvko7G77zuOmPssfQx_B70umFZTD5MxNY0pYcEojvh1BOnvFEZWmfy21HECgmlkgnY0gmlwhKwSclWKbXVsdGlhZGRyc5YACASsEnJVBl7bAAoErBJyVQZe3N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDj4HqRGkLx7HJbrvH3U0TwBbcGOMJG1dr5YjgtpNBCDSDdGNwgl7bg3VkcIJe3YV3YWt1MgU"}'
2026-03-14 04:34:02.597 INFO [src.node.waku_node] REST service is ready !!
2026-03-14 04:34:02.604 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-14 04:34:02.604 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a__wakuorg_nwaku:latest.log
2026-03-14 04:34:02.604 DEBUG [src.node.waku_node] Starting Node...
2026-03-14 04:34:02.604 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-14 04:34:02.606 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-14 04:34:02.606 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.13.62
2026-03-14 04:34:02.606 DEBUG [src.node.docker_mananger] Generated ports ['41098', '41099', '41100', '41101', '41102']
2026-03-14 04:34:02.606 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-14 04:34:02.606 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-14 04:34:02.607 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-14 04:34:02.607 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-14 04:34:02.607 DEBUG [src.node.waku_node] Using volumes []
2026-03-14 04:34:02.607 DEBUG [src.node.docker_mananger] docker run -i -t -p 41098:41098 -p 41099:41099 -p 41100:41100 -p 41101:41101 -p 41102:41102 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=41100 --rest-port=41098 --tcp-port=41099 --discv5-udp-port=41101 --rest-address=0.0.0.0 --nat=extip:172.18.13.62 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=2d6a8e52ef3cb1ceb25f9ac9a1f2a61ac1acde43c26af99bffd4ccadbcba0f53 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=41102 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QID5RdCxiMY3RrQwjv0W7x_0IDjUcsvko7G77zuOmPssfQx_B70umFZTD5MxNY0pYcEojvh1BOnvFEZWmfy21HECgmlkgnY0gmlwhKwSclWKbXVsdGlhZGRyc5YACASsEnJVBl7bAAoErBJyVQZe3N0DgnJzhQACAQAAiXNlY3AyNTZrMaEDj4HqRGkLx7HJbrvH3U0TwBbcGOMJG1dr5YjgtpNBCDSDdGNwgl7bg3VkcIJe3YV3YWt1MgU --content-topic=/toychat/2/huilong/proto
2026-03-14 04:34:02.799 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.13.62 waku 9415f3774c7d365bdc91630b36d5446ff116ea78edb879788aaf89825776daea
2026-03-14 04:34:02.837 DEBUG [src.node.docker_mananger] Container started with ID 9415f3774c7d. Setting up logs at ./log/docker/node2_2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a__wakuorg_nwaku:latest.log
2026-03-14 04:34:02.838 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 41098
2026-03-14 04:34:02.839 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-14 04:34:03.839 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41098/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:03.842 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:03.942 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41098/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:03.945 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:04.045 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41098/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:04.048 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-14 04:34:04.048 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-14 04:34:04.048 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41098/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:04.051 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.13.62/tcp/41099/p2p/16Uiu2HAm19RFT1M7z4Ri5S7hR5V6xFkGjyXDUHyN6KiwhHH1hPr5","/ip4/172.18.13.62/tcp/41100/ws/p2p/16Uiu2HAm19RFT1M7z4Ri5S7hR5V6xFkGjyXDUHyN6KiwhHH1hPr5"],"enrUri":"enr:-L24QI_gzjiuMJIS7OIlXSS8uLV88op-n9KP3TRE13p1We2nBr7MLNqkyP_ZMvwKtVRpAK1053LFGw5ZP4g9JZLwPzYCgmlkgnY0gmlwhKwSDT6KbXVsdGlhZGRyc5YACASsEg0-BqCLAAoErBINPgagjN0DgnJzhQACAQAAiXNlY3AyNTZrMaECVPwsWxELoFdkerZwOkw1rus-BpSE7gEpehCBdwWKRHaDdGNwgqCLg3VkcIKgjYV3YWt1MgE"}'
2026-03-14 04:34:04.051 INFO [src.node.waku_node] REST service is ready !!
2026-03-14 04:34:04.051 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41098/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.114.85/tcp/24283/p2p/16Uiu2HAmNKBYo3XJg3iBwPQsFr8bjpNc25MKAqHXGbkXqhQnCR9V"]'
2026-03-14 04:34:04.076 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:04.078 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:24282/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-14 04:34:04.087 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:04.088 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41098/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-14 04:34:04.093 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:04.094 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:24282/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-14 04:34:04.103 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:04.103 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-14 04:34:04.205 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-14 04:34:04.206 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24282/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:04.214 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773462840595462317,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773462844094627741,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773462844185779513,"ephemeral":false,"proof":""}]'
2026-03-14 04:34:04.259 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-14 04:34:04.260 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-14 04:34:04.260 DEBUG [src.node.waku_node] Stopping container with id b05322eda0a6
2026-03-14 04:34:04.845 DEBUG [src.node.waku_node] Container stopped.
2026-03-14 04:34:04.845 DEBUG [src.node.waku_node] Stopping container with id 9415f3774c7d
2026-03-14 04:34:05.426 DEBUG [src.node.waku_node] Container stopped.
2026-03-14 04:34:05.429 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-14 04:34:05.469 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-14 04:34:05.501 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-14 04:34:05.504 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-14 04:34:05.504 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-14 04:34:05.505 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a__wakuorg_nwaku:latest.log
2026-03-14 04:34:05.508 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-14_04-34-01__f070e84a-a5c1-4cfd-a642-4370aec4dd8a__wakuorg_nwaku:latest.log
2026-03-14 04:34:05.517 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-14 04:34:05.518 DEBUG [tests.conftest] Running test: test_content_topic_also_in_docker_flags[/toychat/2/huilong/proto] with id: 2026-03-14_04-34-05__bea2a6c4-45ca-40eb-9b2a-820a0ca195a9
2026-03-14 04:34:05.520 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-14 04:34:05.521 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-14 04:34:05.521 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-14 04:34:05.533 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-14 04:34:05.535 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-14_04-34-05__bea2a6c4-45ca-40eb-9b2a-820a0ca195a9__wakuorg_nwaku:latest.log
2026-03-14 04:34:05.535 DEBUG [src.node.waku_node] Starting Node...
2026-03-14 04:34:05.535 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-14 04:34:05.538 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-14 04:34:05.539 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.115.151
2026-03-14 04:34:05.539 DEBUG [src.node.docker_mananger] Generated ports ['54420', '54421', '54422', '54423', '54424']
2026-03-14 04:34:05.540 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-14 04:34:05.540 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-14 04:34:05.540 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-14 04:34:05.541 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-14 04:34:05.542 DEBUG [src.node.waku_node] Using volumes []
2026-03-14 04:34:05.542 DEBUG [src.node.docker_mananger] docker run -i -t -p 54420:54420 -p 54421:54421 -p 54422:54422 -p 54423:54423 -p 54424:54424 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=54422 --rest-port=54420 --tcp-port=54421 --discv5-udp-port=54423 --rest-address=0.0.0.0 --nat=extip:172.18.115.151 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=34ef89769bcdbeec8cffe89a04ccb3c9a33893c23d078e7ea4fc0475e187afba --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=54424 --metrics-logging=true --relay=true --filter=true --content-topic=/toychat/2/huilong/proto
2026-03-14 04:34:05.705 ERROR [src.node.docker_mananger] Max retries reached for container b05322eda0a6. Exiting log stream.
2026-03-14 04:34:05.743 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.115.151 waku cba4cecf507cca04ae30109e289f5e3dd41ee8877acdd785b8624d9af3c62f52
2026-03-14 04:34:05.781 DEBUG [src.node.docker_mananger] Container started with ID cba4cecf507c. Setting up logs at ./log/docker/node1_2026-03-14_04-34-05__bea2a6c4-45ca-40eb-9b2a-820a0ca195a9__wakuorg_nwaku:latest.log
2026-03-14 04:34:05.782 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 54420
2026-03-14 04:34:05.782 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-14 04:34:06.435 ERROR [src.node.docker_mananger] Max retries reached for container 9415f3774c7d. Exiting log stream.
2026-03-14 04:34:06.783 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:06.786 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:06.887 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:06.890 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:06.990 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:06.993 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:07.093 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:07.096 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:07.197 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:07.199 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:07.299 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:07.302 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:07.403 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:07.407 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-14 04:34:07.407 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-14 04:34:07.408 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:07.412 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.115.151/tcp/54421/p2p/16Uiu2HAkye192pVKgiCEgufqDA6LTQJjaPZ2AtQSW913aR5HFTtF","/ip4/172.18.115.151/tcp/54422/ws/p2p/16Uiu2HAkye192pVKgiCEgufqDA6LTQJjaPZ2AtQSW913aR5HFTtF"],"enrUri":"enr:-L24QJ8k7PCh7ejpWqBaRR6Jpm8A1SoGzm06dWEFwXzP3r8WEa814Ju2mEuYtRsCv7myXmyNf5CVd11hkvXQA0nw-xcCgmlkgnY0gmlwhKwSc5eKbXVsdGlhZGRyc5YACASsEnOXBtSVAAoErBJzlwbUlt0DgnJzhQACAQAAiXNlY3AyNTZrMaECPpdhgffZ_U58wp6gosrsA0H2b0JmlYA4zkUpZZcCYXSDdGNwgtSVg3VkcILUl4V3YWt1MgU"}'
2026-03-14 04:34:07.412 INFO [src.node.waku_node] REST service is ready !!
2026-03-14 04:34:07.419 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-14 04:34:07.420 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-14_04-34-05__bea2a6c4-45ca-40eb-9b2a-820a0ca195a9__wakuorg_nwaku:latest.log
2026-03-14 04:34:07.420 DEBUG [src.node.waku_node] Starting Node...
2026-03-14 04:34:07.420 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-14 04:34:07.421 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-14 04:34:07.422 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.236.214
2026-03-14 04:34:07.422 DEBUG [src.node.docker_mananger] Generated ports ['45746', '45747', '45748', '45749', '45750']
2026-03-14 04:34:07.422 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-14 04:34:07.422 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-14 04:34:07.422 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-14 04:34:07.422 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-14 04:34:07.422 DEBUG [src.node.waku_node] Using volumes []
2026-03-14 04:34:07.422 DEBUG [src.node.docker_mananger] docker run -i -t -p 45746:45746 -p 45747:45747 -p 45748:45748 -p 45749:45749 -p 45750:45750 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=45748 --rest-port=45746 --tcp-port=45747 --discv5-udp-port=45749 --rest-address=0.0.0.0 --nat=extip:172.18.236.214 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=3c1d284abd75b86a9d49a733b367fa5ede65cdb354fb5b5652c1dccc456aff73 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=45750 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJ8k7PCh7ejpWqBaRR6Jpm8A1SoGzm06dWEFwXzP3r8WEa814Ju2mEuYtRsCv7myXmyNf5CVd11hkvXQA0nw-xcCgmlkgnY0gmlwhKwSc5eKbXVsdGlhZGRyc5YACASsEnOXBtSVAAoErBJzlwbUlt0DgnJzhQACAQAAiXNlY3AyNTZrMaECPpdhgffZ_U58wp6gosrsA0H2b0JmlYA4zkUpZZcCYXSDdGNwgtSVg3VkcILUl4V3YWt1MgU --content-topic=/toychat/2/huilong/proto
2026-03-14 04:34:07.629 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.236.214 waku 791eda618abb4ca21b0c81b2cd0a931650b1be162cc6411aee6d856eb3eeba92
2026-03-14 04:34:07.668 DEBUG [src.node.docker_mananger] Container started with ID 791eda618abb. Setting up logs at ./log/docker/node2_2026-03-14_04-34-05__bea2a6c4-45ca-40eb-9b2a-820a0ca195a9__wakuorg_nwaku:latest.log
2026-03-14 04:34:07.668 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 45746
2026-03-14 04:34:07.668 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-14 04:34:08.669 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:08.671 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:08.772 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:08.775 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:08.875 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:08.878 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:08.978 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:08.981 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:09.081 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.084 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:09.185 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.187 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:09.288 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.291 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:09.391 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.394 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-14 04:34:09.494 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/health" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.497 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-14 04:34:09.497 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-14 04:34:09.497 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.500 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.236.214/tcp/45747/p2p/16Uiu2HAkyp8UNj9NL4bkxGcokp1nETpghqmd6kGcnT9gx6fyGA2A","/ip4/172.18.236.214/tcp/45748/ws/p2p/16Uiu2HAkyp8UNj9NL4bkxGcokp1nETpghqmd6kGcnT9gx6fyGA2A"],"enrUri":"enr:-L24QEjjJ98dGCe-jv6XiqmNv-bKjfdXBi2xSjlNH3U1Dpr9VHayBdxzxCPg7McA1eRz32ZR1RhG6lXsptDdPaoqBroCgmlkgnY0gmlwhKwS7NaKbXVsdGlhZGRyc5YACASsEuzWBrKzAAoErBLs1gaytN0DgnJzhQACAQAAiXNlY3AyNTZrMaECQS97IsvkWPpxCRU6i9AImNGtTEZ_kZQHfuUQFm8gJ9-DdGNwgrKzg3VkcIKytYV3YWt1MgE"}'
2026-03-14 04:34:09.500 INFO [src.node.waku_node] REST service is ready !!
2026-03-14 04:34:09.501 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45746/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.115.151/tcp/54421/p2p/16Uiu2HAkye192pVKgiCEgufqDA6LTQJjaPZ2AtQSW913aR5HFTtF"]'
2026-03-14 04:34:09.523 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:09.524 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54420/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-14 04:34:09.536 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:09.538 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45746/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-14 04:34:09.545 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:09.548 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54420/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-14 04:34:09.560 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-14 04:34:09.561 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-14 04:34:09.661 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-14 04:34:09.662 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54420/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.665 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773462849547790257,"ephemeral":false,"proof":""}]'
2026-03-14 04:34:09.667 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-14 04:34:09.667 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45746/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-14 04:34:09.669 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773462849547790257,"ephemeral":false,"proof":""}]'
2026-03-14 04:34:09.673 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-14 04:34:09.674 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-14 04:34:09.674 DEBUG [src.node.waku_node] Stopping container with id cba4cecf507c
2026-03-14 04:34:10.287 DEBUG [src.node.waku_node] Container stopped.
2026-03-14 04:34:10.288 DEBUG [src.node.waku_node] Stopping container with id 791eda618abb
2026-03-14 04:34:10.915 DEBUG [src.node.waku_node] Container stopped.
2026-03-14 04:34:10.917 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-14 04:34:10.951 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-14 04:34:10.982 DEBUG [src.node.docker_mananger] No errors found in the waku logs.