208 lines
36 KiB
Plaintext

2026-03-17 04:36:41.317 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-17 04:36:41.317 DEBUG [tests.conftest] Running test: test_subscribe_and_publish_on_another_content_topic_from_another_shard with id: 2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d
2026-03-17 04:36:41.317 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-17 04:36:41.318 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-17 04:36:41.318 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-17 04:36:41.325 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:36:41.325 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d__wakuorg_nwaku:latest.log
2026-03-17 04:36:41.326 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:36:41.326 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:36:41.327 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:36:41.327 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.28.130
2026-03-17 04:36:41.327 DEBUG [src.node.docker_mananger] Generated ports ['30420', '30421', '30422', '30423', '30424']
2026-03-17 04:36:41.327 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:36:41.328 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:36:41.328 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:36:41.328 DEBUG [src.node.docker_mananger] docker run -i -t -p 30420:30420 -p 30421:30421 -p 30422:30422 -p 30423:30423 -p 30424:30424 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=30422 --rest-port=30420 --tcp-port=30421 --discv5-udp-port=30423 --rest-address=0.0.0.0 --nat=extip:172.18.28.130 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=ebe4bbaecbcf36bbeeceaf281c37b69d25f9b2ff0b6d1f8fda49dfaabc1adb8c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=30424 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-17 04:36:41.528 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.28.130 waku 83c3759c3ba7e51ccb11a91b4f79f4a5ba5950dbeba1f11cda913ee57e8c2d28
2026-03-17 04:36:41.566 DEBUG [src.node.docker_mananger] Container started with ID 83c3759c3ba7. Setting up logs at ./log/docker/node1_2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d__wakuorg_nwaku:latest.log
2026-03-17 04:36:41.568 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 30420
2026-03-17 04:36:41.570 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:36:42.123 ERROR [src.node.docker_mananger] Max retries reached for container 06791ada7bdb. Exiting log stream.
2026-03-17 04:36:42.571 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30420/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:42.575 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:42.676 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30420/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:42.679 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:36:42.679 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:36:42.679 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30420/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:42.682 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.28.130/tcp/30421/p2p/16Uiu2HAmGyGDF6R73XRPpGhJtCyUS4im72uxGjdur7zAcbBxxkVQ","/ip4/172.18.28.130/tcp/30422/ws/p2p/16Uiu2HAmGyGDF6R73XRPpGhJtCyUS4im72uxGjdur7zAcbBxxkVQ"],"enrUri":"enr:-L24QJ81RVCKkHJACk1TPNLUqbP3d8Qm1pujZU7QGOGE3eprN8VJePock6wQSut241EaFoBI6NrCGuc1ee_Nv5usigMCgmlkgnY0gmlwhKwSHIKKbXVsdGlhZGRyc5YACASsEhyCBnbVAAoErBIcggZ21t0DgnJzhQACAQAAiXNlY3AyNTZrMaEDQB0i_MBYrzTu1vLxAERAqPMpVo031SMA6LfA8r0Lt8ODdGNwgnbVg3VkcIJ214V3YWt1MgU"}'
2026-03-17 04:36:42.682 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:36:42.689 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:36:42.689 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d__wakuorg_nwaku:latest.log
2026-03-17 04:36:42.690 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:36:42.690 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:36:42.691 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:36:42.691 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.20.38
2026-03-17 04:36:42.691 DEBUG [src.node.docker_mananger] Generated ports ['14981', '14982', '14983', '14984', '14985']
2026-03-17 04:36:42.691 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:36:42.692 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:36:42.692 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:36:42.692 DEBUG [src.node.docker_mananger] docker run -i -t -p 14981:14981 -p 14982:14982 -p 14983:14983 -p 14984:14984 -p 14985:14985 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=14983 --rest-port=14981 --tcp-port=14982 --discv5-udp-port=14984 --rest-address=0.0.0.0 --nat=extip:172.18.20.38 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=2d7b4e9793f43a0f8be2bb8efefabdf111e28d6a9ccbed9bbe5ec9248eaeaece --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=14985 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJ81RVCKkHJACk1TPNLUqbP3d8Qm1pujZU7QGOGE3eprN8VJePock6wQSut241EaFoBI6NrCGuc1ee_Nv5usigMCgmlkgnY0gmlwhKwSHIKKbXVsdGlhZGRyc5YACASsEhyCBnbVAAoErBIcggZ21t0DgnJzhQACAQAAiXNlY3AyNTZrMaEDQB0i_MBYrzTu1vLxAERAqPMpVo031SMA6LfA8r0Lt8ODdGNwgnbVg3VkcIJ214V3YWt1MgU --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-17 04:36:42.904 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.20.38 waku cfd9bce53175151ef2aea7c38e422aa55a898801ff1cdfce4eb3a0ec3cab5b23
2026-03-17 04:36:42.939 DEBUG [src.node.docker_mananger] Container started with ID cfd9bce53175. Setting up logs at ./log/docker/node2_2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d__wakuorg_nwaku:latest.log
2026-03-17 04:36:42.939 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 14981
2026-03-17 04:36:42.939 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:36:43.940 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:43.942 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.043 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.045 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.146 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.148 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.249 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.252 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.352 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.355 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.455 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.458 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.558 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.561 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.661 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.664 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.764 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.767 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.867 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.870 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:44.970 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.973 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:36:44.973 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:36:44.973 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:44.975 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.20.38/tcp/14982/p2p/16Uiu2HAmKvxH8kBbSjq7jrtGEXgvAB12vkHLT2CgbwZo5issA33C","/ip4/172.18.20.38/tcp/14983/ws/p2p/16Uiu2HAmKvxH8kBbSjq7jrtGEXgvAB12vkHLT2CgbwZo5issA33C"],"enrUri":"enr:-L24QL6JgAeinKK-qd60IOC9I2W2Q4Otj5boVaXcBydydX0gLSlGzW8UEzRiLIgHZiuoi2NMUUY-OJFlggtqrzmoeaUCgmlkgnY0gmlwhKwSFCaKbXVsdGlhZGRyc5YACASsEhQmBjqGAAoErBIUJgY6h90DgnJzhQACAQAAiXNlY3AyNTZrMaEDbBjCTogL6wC5Kj1nRFnkxYvbRQXZcp28tRYI8tsLnM-DdGNwgjqGg3VkcII6iIV3YWt1MgE"}'
2026-03-17 04:36:44.976 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:36:44.976 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:14981/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.28.130/tcp/30421/p2p/16Uiu2HAmGyGDF6R73XRPpGhJtCyUS4im72uxGjdur7zAcbBxxkVQ"]'
2026-03-17 04:36:45.006 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:45.008 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30420/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-17 04:36:45.012 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:45.014 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:14981/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-17 04:36:45.020 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:45.020 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30420/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-17 04:36:45.030 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:45.031 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-17 04:36:45.131 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-17 04:36:45.132 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30420/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:45.134 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773722205020731479,"ephemeral":false,"proof":""}]'
2026-03-17 04:36:45.136 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-17 04:36:45.137 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14981/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:45.140 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773722205020731479,"ephemeral":false,"proof":""}]'
2026-03-17 04:36:45.142 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30420/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-17 04:36:45.150 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:45.151 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-17 04:36:45.252 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-17 04:36:45.252 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30420/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:45.254 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773722203050865063,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773722201866925156,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773722205141971000,"ephemeral":false,"proof":""}]'
2026-03-17 04:36:45.296 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-17 04:36:45.297 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-17 04:36:45.297 DEBUG [src.node.waku_node] Stopping container with id 83c3759c3ba7
2026-03-17 04:36:45.908 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:36:45.909 DEBUG [src.node.waku_node] Stopping container with id cfd9bce53175
2026-03-17 04:36:46.521 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:36:46.524 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-17 04:36:46.576 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:36:46.607 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:36:46.609 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-17 04:36:46.609 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-17 04:36:46.611 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d__wakuorg_nwaku:latest.log
2026-03-17 04:36:46.612 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-17_04-36-41__cfdd92b5-a4c0-4ae2-bc0d-ea3e4385bd1d__wakuorg_nwaku:latest.log
2026-03-17 04:36:46.627 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-17 04:36:46.628 DEBUG [tests.conftest] Running test: test_subscribe_and_publish_on_another_content_topic_from_another_shard with id: 2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a
2026-03-17 04:36:46.628 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-17 04:36:46.628 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-17 04:36:46.629 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-17 04:36:46.641 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:36:46.642 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a__wakuorg_nwaku:latest.log
2026-03-17 04:36:46.643 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:36:46.643 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:36:46.646 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:36:46.646 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.174.46
2026-03-17 04:36:46.647 DEBUG [src.node.docker_mananger] Generated ports ['3456', '3457', '3458', '3459', '3460']
2026-03-17 04:36:46.648 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:36:46.648 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:36:46.649 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:36:46.650 DEBUG [src.node.docker_mananger] docker run -i -t -p 3456:3456 -p 3457:3457 -p 3458:3458 -p 3459:3459 -p 3460:3460 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=3458 --rest-port=3456 --tcp-port=3457 --discv5-udp-port=3459 --rest-address=0.0.0.0 --nat=extip:172.18.174.46 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=cdead92c2afcadc8bca52fbdfef6dcbebe0af2a6ca22bbadfdfac1a1bc9fcd7d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=3460 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-17 04:36:46.780 ERROR [src.node.docker_mananger] Max retries reached for container 83c3759c3ba7. Exiting log stream.
2026-03-17 04:36:46.874 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.174.46 waku 1e52d1b37231638e4290238797a405defbb7c1e6de929c6b2a32fa3e18faa935
2026-03-17 04:36:46.914 DEBUG [src.node.docker_mananger] Container started with ID 1e52d1b37231. Setting up logs at ./log/docker/node1_2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a__wakuorg_nwaku:latest.log
2026-03-17 04:36:46.914 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 3456
2026-03-17 04:36:46.914 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:36:47.557 ERROR [src.node.docker_mananger] Max retries reached for container cfd9bce53175. Exiting log stream.
2026-03-17 04:36:47.915 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3456/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:47.918 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:48.019 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3456/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:48.021 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:36:48.021 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:36:48.022 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3456/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:48.024 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.174.46/tcp/3457/p2p/16Uiu2HAm6RsjCsjY9XrtdWupj9UkiP91tLmFnfLEyv48hH22PKbo","/ip4/172.18.174.46/tcp/3458/ws/p2p/16Uiu2HAm6RsjCsjY9XrtdWupj9UkiP91tLmFnfLEyv48hH22PKbo"],"enrUri":"enr:-L24QHOlP_KKPA2jOMRF0v5bL2wl7IBkL_ozV0Af_21yNXDtO1p-tKtVW8N-U9q4UQqVj_0HgRz7oTrgdaD7R1f4fH4CgmlkgnY0gmlwhKwSri6KbXVsdGlhZGRyc5YACASsEq4uBg2BAAoErBKuLgYNgt0DgnJzhQACAQAAiXNlY3AyNTZrMaECo33XGh_l6BRZj0kXyN49ML4J02ZP7Yogkc09YTPfl4qDdGNwgg2Bg3VkcIINg4V3YWt1MgU"}'
2026-03-17 04:36:48.024 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:36:48.032 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-17 04:36:48.032 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a__wakuorg_nwaku:latest.log
2026-03-17 04:36:48.032 DEBUG [src.node.waku_node] Starting Node...
2026-03-17 04:36:48.032 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-17 04:36:48.033 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-17 04:36:48.034 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.60.125
2026-03-17 04:36:48.034 DEBUG [src.node.docker_mananger] Generated ports ['16995', '16996', '16997', '16998', '16999']
2026-03-17 04:36:48.034 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-17 04:36:48.034 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-17 04:36:48.034 DEBUG [src.node.waku_node] Using volumes []
2026-03-17 04:36:48.034 DEBUG [src.node.docker_mananger] docker run -i -t -p 16995:16995 -p 16996:16996 -p 16997:16997 -p 16998:16998 -p 16999:16999 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=16997 --rest-port=16995 --tcp-port=16996 --discv5-udp-port=16998 --rest-address=0.0.0.0 --nat=extip:172.18.60.125 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=33ef0ec56372b7da98c6ac1587dd5d60ebbdbf91fbe13adcb9f48cf07dce0fb6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=16999 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHOlP_KKPA2jOMRF0v5bL2wl7IBkL_ozV0Af_21yNXDtO1p-tKtVW8N-U9q4UQqVj_0HgRz7oTrgdaD7R1f4fH4CgmlkgnY0gmlwhKwSri6KbXVsdGlhZGRyc5YACASsEq4uBg2BAAoErBKuLgYNgt0DgnJzhQACAQAAiXNlY3AyNTZrMaECo33XGh_l6BRZj0kXyN49ML4J02ZP7Yogkc09YTPfl4qDdGNwgg2Bg3VkcIINg4V3YWt1MgU --content-topic=/myapp/1/latest/proto --num-shards-in-network=8
2026-03-17 04:36:48.242 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.60.125 waku 02a1b5b8dfa6cf1998bd1246038e2b73ca326db6c9329ed85a005a931f7eba57
2026-03-17 04:36:48.287 DEBUG [src.node.docker_mananger] Container started with ID 02a1b5b8dfa6. Setting up logs at ./log/docker/node2_2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a__wakuorg_nwaku:latest.log
2026-03-17 04:36:48.287 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 16995
2026-03-17 04:36:48.287 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-17 04:36:49.287 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.290 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:49.391 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.393 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:49.494 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.497 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:49.597 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.600 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:49.700 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.702 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:49.803 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.805 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:49.905 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:49.908 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:50.008 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.011 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:50.112 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.133 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-17 04:36:50.234 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/health" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.237 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-17 04:36:50.237 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-17 04:36:50.237 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.239 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.60.125/tcp/16996/p2p/16Uiu2HAmCCwRxZh9zP8rAKd4A9uy4unHDDb9W4FS4GwNpgPNxPFd","/ip4/172.18.60.125/tcp/16997/ws/p2p/16Uiu2HAmCCwRxZh9zP8rAKd4A9uy4unHDDb9W4FS4GwNpgPNxPFd"],"enrUri":"enr:-L24QIVDGYDGZwt3cWu08Y6SZ3R6rKU9Izh9PR3OMWi7h1kVXSHnNuP5uaPsX8CJ-THnJjUGLEuQIfmtlCqZBtnSLi8CgmlkgnY0gmlwhKwSPH2KbXVsdGlhZGRyc5YACASsEjx9BkJkAAoErBI8fQZCZd0DgnJzhQACAQAAiXNlY3AyNTZrMaEC-VOZDj5_VGqpEgUfqB5ZW2Er7vZYnmC_sDxoAgWoZ1CDdGNwgkJkg3VkcIJCZoV3YWt1MgE"}'
2026-03-17 04:36:50.240 INFO [src.node.waku_node] REST service is ready !!
2026-03-17 04:36:50.240 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16995/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.174.46/tcp/3457/p2p/16Uiu2HAm6RsjCsjY9XrtdWupj9UkiP91tLmFnfLEyv48hH22PKbo"]'
2026-03-17 04:36:50.264 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:50.267 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3456/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-17 04:36:50.274 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:50.275 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16995/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/toychat/2/huilong/proto"]'
2026-03-17 04:36:50.282 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:50.283 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3456/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/toychat/2/huilong/proto", "timestamp": '$(date +%s%N)'}'
2026-03-17 04:36:50.297 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:50.299 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-17 04:36:50.400 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-17 04:36:50.400 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3456/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.403 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773722210283245365,"ephemeral":false,"proof":""}]'
2026-03-17 04:36:50.404 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-17 04:36:50.404 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16995/relay/v1/auto/messages/%2Ftoychat%2F2%2Fhuilong%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.407 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/toychat/2/huilong/proto","version":0,"timestamp":1773722210283245365,"ephemeral":false,"proof":""}]'
2026-03-17 04:36:50.408 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3456/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-17 04:36:50.416 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-17 04:36:50.417 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-17 04:36:50.518 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-17 04:36:50.518 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3456/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-17 04:36:50.521 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773722208226783803,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773722209399113300,"ephemeral":false,"proof":""},{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773722210408529058,"ephemeral":false,"proof":""}]'
2026-03-17 04:36:50.550 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-17 04:36:50.551 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-17 04:36:50.551 DEBUG [src.node.waku_node] Stopping container with id 1e52d1b37231
2026-03-17 04:36:51.173 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:36:51.175 DEBUG [src.node.waku_node] Stopping container with id 02a1b5b8dfa6
2026-03-17 04:36:51.741 DEBUG [src.node.waku_node] Container stopped.
2026-03-17 04:36:51.743 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-17 04:36:51.789 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:36:51.821 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-17 04:36:51.822 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-17 04:36:51.822 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-17 04:36:51.822 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a__wakuorg_nwaku:latest.log
2026-03-17 04:36:51.823 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-17_04-36-46__8dd87ef6-7f74-4c63-ae90-5b39b92c357a__wakuorg_nwaku:latest.log