200 lines
34 KiB
Plaintext

2026-03-07 04:39:17.938 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-07 04:39:17.938 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c
2026-03-07 04:39:17.939 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-07 04:39:17.940 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-07 04:39:17.940 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-07 04:39:17.952 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:39:17.954 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:17.954 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:39:17.954 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:39:17.957 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:39:17.957 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.11.38
2026-03-07 04:39:17.957 DEBUG [src.node.docker_mananger] Generated ports ['54502', '54503', '54504', '54505', '54506']
2026-03-07 04:39:17.958 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-07 04:39:17.958 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-07 04:39:17.958 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:39:17.958 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:39:17.958 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:39:17.958 DEBUG [src.node.docker_mananger] docker run -i -t -p 54502:54502 -p 54503:54503 -p 54504:54504 -p 54505:54505 -p 54506:54506 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=54504 --rest-port=54502 --tcp-port=54503 --discv5-udp-port=54505 --rest-address=0.0.0.0 --nat=extip:172.18.11.38 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=5e42b85fda51c8e3218d683f2de54e3ab96e2502db7e02321e540d5245facb9c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=54506 --metrics-logging=true --relay=true --filter=true
2026-03-07 04:39:18.158 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.11.38 waku 38c19e99b2826ceccd1edc20f210aa77a05bf76eaa60542a8527647d8f62f884
2026-03-07 04:39:18.161 ERROR [src.node.docker_mananger] Max retries reached for container 6cd0aaa89e61. Exiting log stream.
2026-03-07 04:39:18.202 DEBUG [src.node.docker_mananger] Container started with ID 38c19e99b282. Setting up logs at ./log/docker/node1_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:18.202 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 54502
2026-03-07 04:39:18.202 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:39:18.765 ERROR [src.node.docker_mananger] Max retries reached for container 55903b937de9. Exiting log stream.
2026-03-07 04:39:19.203 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54502/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:19.205 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:19.306 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54502/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:19.308 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:19.409 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54502/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:19.411 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:39:19.411 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:39:19.412 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54502/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:19.414 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.11.38/tcp/54503/p2p/16Uiu2HAm57JSU4dRXsvsHsN5bCcnXSbuGaFkiQFT6LB8Vfdrs8QA","/ip4/172.18.11.38/tcp/54504/ws/p2p/16Uiu2HAm57JSU4dRXsvsHsN5bCcnXSbuGaFkiQFT6LB8Vfdrs8QA"],"enrUri":"enr:-L24QFDFBU5mPsT_HZ-nIOHm6RNGOAEAkfGGuB3SgIyT8tFyICrlJW5BbWa1QitNGrbRZl9a92uqyePMcrVhKedG_QQCgmlkgnY0gmlwhKwSCyaKbXVsdGlhZGRyc5YACASsEgsmBtTnAAoErBILJgbU6N0DgnJzhQACAQAAiXNlY3AyNTZrMaECj-APKWc2P5MpPrXQffIKBnbTXrNKEGL9yC0i29Ep-huDdGNwgtTng3VkcILU6YV3YWt1MgU"}'
2026-03-07 04:39:19.414 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:39:19.421 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:39:19.422 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:19.422 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:39:19.422 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:39:19.423 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:39:19.423 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.139.198
2026-03-07 04:39:19.424 DEBUG [src.node.docker_mananger] Generated ports ['2179', '2180', '2181', '2182', '2183']
2026-03-07 04:39:19.424 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-07 04:39:19.424 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-07 04:39:19.424 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:39:19.424 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:39:19.424 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:39:19.424 DEBUG [src.node.docker_mananger] docker run -i -t -p 2179:2179 -p 2180:2180 -p 2181:2181 -p 2182:2182 -p 2183:2183 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2181 --rest-port=2179 --tcp-port=2180 --discv5-udp-port=2182 --rest-address=0.0.0.0 --nat=extip:172.18.139.198 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=0a83e6e3510bb569bdbb668ff5ad58055ad4fcfcfc5ca24d9c72fdba24158c1a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2183 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFDFBU5mPsT_HZ-nIOHm6RNGOAEAkfGGuB3SgIyT8tFyICrlJW5BbWa1QitNGrbRZl9a92uqyePMcrVhKedG_QQCgmlkgnY0gmlwhKwSCyaKbXVsdGlhZGRyc5YACASsEgsmBtTnAAoErBILJgbU6N0DgnJzhQACAQAAiXNlY3AyNTZrMaECj-APKWc2P5MpPrXQffIKBnbTXrNKEGL9yC0i29Ep-huDdGNwgtTng3VkcILU6YV3YWt1MgU
2026-03-07 04:39:19.649 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.139.198 waku 9108cd4c3a7e40e9833350a48e3a3bd979c062157772fde914d359cb70b4cfa0
2026-03-07 04:39:19.684 DEBUG [src.node.docker_mananger] Container started with ID 9108cd4c3a7e. Setting up logs at ./log/docker/node2_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:19.685 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2179
2026-03-07 04:39:19.685 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:39:20.685 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:20.688 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:20.788 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:20.790 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:20.891 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:20.893 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:20.993 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:20.996 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:21.096 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:21.099 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:21.199 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:21.202 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:21.302 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:21.305 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:21.406 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:21.410 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-07 04:39:21.510 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:21.513 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:39:21.513 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:39:21.513 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:21.515 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.139.198/tcp/2180/p2p/16Uiu2HAmPEoi2r3JuitZb4NSUXfBBXhFgTb4F2YwXxnmCnwrkaLV","/ip4/172.18.139.198/tcp/2181/ws/p2p/16Uiu2HAmPEoi2r3JuitZb4NSUXfBBXhFgTb4F2YwXxnmCnwrkaLV"],"enrUri":"enr:-L24QNH1g4ipvbyplR5kSnU6Y0AWoPFrTgsb211iQFu269LJcEhsl5p58Kx0MYoPJ5yhmS608Un121mlnU8NrFMDl60CgmlkgnY0gmlwhKwSi8aKbXVsdGlhZGRyc5YACASsEovGBgiEAAoErBKLxgYIhd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDnT6VWl-BJI2rEwJzM832fRyJtIXhxH8TEvZ8UQF4QVaDdGNwggiEg3VkcIIIhoV3YWt1MgE"}'
2026-03-07 04:39:21.516 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:39:21.516 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2179/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.11.38/tcp/54503/p2p/16Uiu2HAm57JSU4dRXsvsHsN5bCcnXSbuGaFkiQFT6LB8Vfdrs8QA"]'
2026-03-07 04:39:21.542 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:21.553 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:39:21.554 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:21.554 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:39:21.555 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:39:21.557 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:39:21.558 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.107.102
2026-03-07 04:39:21.558 DEBUG [src.node.docker_mananger] Generated ports ['56697', '56698', '56699', '56700', '56701']
2026-03-07 04:39:21.558 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-07 04:39:21.558 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-07 04:39:21.558 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:39:21.558 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:39:21.558 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:39:21.559 DEBUG [src.node.docker_mananger] docker run -i -t -p 56697:56697 -p 56698:56698 -p 56699:56699 -p 56700:56700 -p 56701:56701 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=56699 --rest-port=56697 --tcp-port=56698 --discv5-udp-port=56700 --rest-address=0.0.0.0 --nat=extip:172.18.107.102 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=4b7b1af0b3beff3787db8d5fce9aad515cce1284d3ce2fefb1bde51bb222cffe --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=56701 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFDFBU5mPsT_HZ-nIOHm6RNGOAEAkfGGuB3SgIyT8tFyICrlJW5BbWa1QitNGrbRZl9a92uqyePMcrVhKedG_QQCgmlkgnY0gmlwhKwSCyaKbXVsdGlhZGRyc5YACASsEgsmBtTnAAoErBILJgbU6N0DgnJzhQACAQAAiXNlY3AyNTZrMaECj-APKWc2P5MpPrXQffIKBnbTXrNKEGL9yC0i29Ep-huDdGNwgtTng3VkcILU6YV3YWt1MgU
2026-03-07 04:39:21.756 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.107.102 waku 1dfbe65d14dda29f74bb1ce36e148d292c0eb07444d1306169a07bf04fbf53eb
2026-03-07 04:39:21.792 DEBUG [src.node.docker_mananger] Container started with ID 1dfbe65d14dd. Setting up logs at ./log/docker/node3_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:21.793 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 56697
2026-03-07 04:39:21.793 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:39:22.793 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56697/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:22.796 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:39:22.796 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:39:22.797 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56697/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:22.799 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.107.102/tcp/56698/p2p/16Uiu2HAmKrvhRZiyyf5WBa3XDZDk8LqzqtFKWcsJu7XqbkYPpzUj","/ip4/172.18.107.102/tcp/56699/ws/p2p/16Uiu2HAmKrvhRZiyyf5WBa3XDZDk8LqzqtFKWcsJu7XqbkYPpzUj"],"enrUri":"enr:-L24QFi1QPxO1u6EOSVcy9LijvfBbSklSUzRlAdfX5cFdL2Ia3HstRKjQMKVf-x64L95OCsMhg7t8-2sKIIB3nV64w8CgmlkgnY0gmlwhKwSa2aKbXVsdGlhZGRyc5YACASsEmtmBt16AAoErBJrZgbde90DgnJzhQACAQAAiXNlY3AyNTZrMaEDaxCl3KV9ZiMGwmBrQ2RMlhuhe4IekobSVvymk5E1VCSDdGNwgt16g3VkcILdfIV3YWt1MgE"}'
2026-03-07 04:39:22.799 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:39:22.799 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:56697/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.11.38/tcp/54503/p2p/16Uiu2HAm57JSU4dRXsvsHsN5bCcnXSbuGaFkiQFT6LB8Vfdrs8QA"]'
2026-03-07 04:39:22.821 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:22.833 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:39:22.834 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:22.834 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:39:22.835 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:39:22.839 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:39:22.840 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.161.156
2026-03-07 04:39:22.840 DEBUG [src.node.docker_mananger] Generated ports ['52703', '52704', '52705', '52706', '52707']
2026-03-07 04:39:22.840 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-07 04:39:22.842 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-07 04:39:22.842 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:39:22.842 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:39:22.843 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:39:22.844 DEBUG [src.node.docker_mananger] docker run -i -t -p 52703:52703 -p 52704:52704 -p 52705:52705 -p 52706:52706 -p 52707:52707 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=52705 --rest-port=52703 --tcp-port=52704 --discv5-udp-port=52706 --rest-address=0.0.0.0 --nat=extip:172.18.161.156 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f2d06af81f3eedbdeddc65c80edfda3c0d9ffee02f00b15f8ab6baddbecaef20 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=52707 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFDFBU5mPsT_HZ-nIOHm6RNGOAEAkfGGuB3SgIyT8tFyICrlJW5BbWa1QitNGrbRZl9a92uqyePMcrVhKedG_QQCgmlkgnY0gmlwhKwSCyaKbXVsdGlhZGRyc5YACASsEgsmBtTnAAoErBILJgbU6N0DgnJzhQACAQAAiXNlY3AyNTZrMaECj-APKWc2P5MpPrXQffIKBnbTXrNKEGL9yC0i29Ep-huDdGNwgtTng3VkcILU6YV3YWt1MgU
2026-03-07 04:39:23.047 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.161.156 waku 595ae61bdd9d0330c02881a2ccae731cc35b47958f182f1278a954daa3ff2896
2026-03-07 04:39:23.083 DEBUG [src.node.docker_mananger] Container started with ID 595ae61bdd9d. Setting up logs at ./log/docker/node4_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:23.083 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 52703
2026-03-07 04:39:23.083 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:39:24.084 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52703/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:24.087 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:39:24.087 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:39:24.087 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52703/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:24.089 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.161.156/tcp/52704/p2p/16Uiu2HAmMt6viC3CEuEpGA7McJnUKShXoYj3SVSRvGxGEQoPgaBm","/ip4/172.18.161.156/tcp/52705/ws/p2p/16Uiu2HAmMt6viC3CEuEpGA7McJnUKShXoYj3SVSRvGxGEQoPgaBm"],"enrUri":"enr:-L24QIco9tOUlawS0tvmEszHrccWUeYvK5xZpw-1pULGIlLhS6r2dgFqCZzVJIUSlA2HfJrTB9OlFkWBzi2OxbmXWjMCgmlkgnY0gmlwhKwSoZyKbXVsdGlhZGRyc5YACASsEqGcBs3gAAoErBKhnAbN4d0DgnJzhQACAQAAiXNlY3AyNTZrMaEDiRUrKokI39NkkarLiHwnKJUt_X4E2rpv6ZBn9Gy7k4yDdGNwgs3gg3VkcILN4oV3YWt1MgE"}'
2026-03-07 04:39:24.089 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:39:24.090 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52703/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.11.38/tcp/54503/p2p/16Uiu2HAm57JSU4dRXsvsHsN5bCcnXSbuGaFkiQFT6LB8Vfdrs8QA"]'
2026-03-07 04:39:24.110 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:24.126 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-07 04:39:24.126 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:24.127 DEBUG [src.node.waku_node] Starting Node...
2026-03-07 04:39:24.127 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-07 04:39:24.131 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-07 04:39:24.133 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.51.100
2026-03-07 04:39:24.133 DEBUG [src.node.docker_mananger] Generated ports ['30644', '30645', '30646', '30647', '30648']
2026-03-07 04:39:24.133 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-07 04:39:24.134 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-07 04:39:24.134 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-07 04:39:24.135 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-07 04:39:24.136 DEBUG [src.node.waku_node] Using volumes []
2026-03-07 04:39:24.136 DEBUG [src.node.docker_mananger] docker run -i -t -p 30644:30644 -p 30645:30645 -p 30646:30646 -p 30647:30647 -p 30648:30648 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=30646 --rest-port=30644 --tcp-port=30645 --discv5-udp-port=30647 --rest-address=0.0.0.0 --nat=extip:172.18.51.100 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=0cbfab9e7ebe3dccfb3a62db57d81f9ad8a26ccdcfc764bf61ec13eafc86d8b9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=30648 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFDFBU5mPsT_HZ-nIOHm6RNGOAEAkfGGuB3SgIyT8tFyICrlJW5BbWa1QitNGrbRZl9a92uqyePMcrVhKedG_QQCgmlkgnY0gmlwhKwSCyaKbXVsdGlhZGRyc5YACASsEgsmBtTnAAoErBILJgbU6N0DgnJzhQACAQAAiXNlY3AyNTZrMaECj-APKWc2P5MpPrXQffIKBnbTXrNKEGL9yC0i29Ep-huDdGNwgtTng3VkcILU6YV3YWt1MgU
2026-03-07 04:39:24.341 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.51.100 waku 9ac2a76af7567dd1e743cb9a234d9cbbacf9a90f1330f1738af2d2da3271317b
2026-03-07 04:39:24.375 DEBUG [src.node.docker_mananger] Container started with ID 9ac2a76af756. Setting up logs at ./log/docker/node5_2026-03-07_04-39-17__e2aa9ae5-dee1-4d92-8a6c-f03eefc9778c__wakuorg_nwaku:latest.log
2026-03-07 04:39:24.375 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 30644
2026-03-07 04:39:24.375 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-07 04:39:25.376 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30644/health" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.379 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-07 04:39:25.379 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-07 04:39:25.379 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30644/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.381 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.51.100/tcp/30645/p2p/16Uiu2HAm9Yo7XwHrzAVw7TFpqtRJTKy2msTHyFAVzXrjtFkuFmax","/ip4/172.18.51.100/tcp/30646/ws/p2p/16Uiu2HAm9Yo7XwHrzAVw7TFpqtRJTKy2msTHyFAVzXrjtFkuFmax"],"enrUri":"enr:-L24QIivLO_GMXa9_vKKeRsHNMJUmZs9W6H3pwAaqA1vpv3jVY1LdGTR_sf39lXHNVuwFJHjK5d07zp8AVrrq3VYg_MCgmlkgnY0gmlwhKwSM2SKbXVsdGlhZGRyc5YACASsEjNkBne1AAoErBIzZAZ3tt0DgnJzhQACAQAAiXNlY3AyNTZrMaEC0da_mypSjddW2IlBEaew7qGSgMXagVi8WvXY6eE79DGDdGNwgne1g3VkcIJ3t4V3YWt1MgE"}'
2026-03-07 04:39:25.382 INFO [src.node.waku_node] REST service is ready !!
2026-03-07 04:39:25.382 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30644/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.11.38/tcp/54503/p2p/16Uiu2HAm57JSU4dRXsvsHsN5bCcnXSbuGaFkiQFT6LB8Vfdrs8QA"]'
2026-03-07 04:39:25.415 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.416 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54502/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-07 04:39:25.422 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.422 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2179/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-07 04:39:25.425 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.425 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:56697/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-07 04:39:25.428 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.428 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:52703/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-07 04:39:25.430 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.431 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30644/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-07 04:39:25.433 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.433 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54502/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-07 04:39:25.445 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-07 04:39:25.445 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-07 04:39:25.546 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-07 04:39:25.546 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54502/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.548 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772858365433616052,"ephemeral":false,"proof":""}]'
2026-03-07 04:39:25.550 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-07 04:39:25.550 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2179/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.552 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772858365433616052,"ephemeral":false,"proof":""}]'
2026-03-07 04:39:25.554 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-07 04:39:25.554 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56697/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.556 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772858365433616052,"ephemeral":false,"proof":""}]'
2026-03-07 04:39:25.558 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-07 04:39:25.558 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:52703/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.560 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772858365433616052,"ephemeral":false,"proof":""}]'
2026-03-07 04:39:25.562 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-07 04:39:25.562 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30644/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-07 04:39:25.564 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772858365433616052,"ephemeral":false,"proof":""}]'
2026-03-07 04:39:25.567 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-07 04:39:25.568 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-07 04:39:25.568 DEBUG [src.node.waku_node] Stopping container with id 38c19e99b282
2026-03-07 04:39:26.224 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:39:26.225 DEBUG [src.node.waku_node] Stopping container with id 9108cd4c3a7e
2026-03-07 04:39:26.849 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:39:26.849 DEBUG [src.node.waku_node] Stopping container with id 1dfbe65d14dd
2026-03-07 04:39:27.191 ERROR [src.node.docker_mananger] Max retries reached for container 38c19e99b282. Exiting log stream.
2026-03-07 04:39:27.448 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:39:27.449 DEBUG [src.node.waku_node] Stopping container with id 595ae61bdd9d
2026-03-07 04:39:27.666 ERROR [src.node.docker_mananger] Max retries reached for container 9108cd4c3a7e. Exiting log stream.
2026-03-07 04:39:28.019 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:39:28.020 DEBUG [src.node.waku_node] Stopping container with id 9ac2a76af756
2026-03-07 04:39:28.382 ERROR [src.node.docker_mananger] Max retries reached for container 1dfbe65d14dd. Exiting log stream.
2026-03-07 04:39:28.597 DEBUG [src.node.waku_node] Container stopped.
2026-03-07 04:39:28.599 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-07 04:39:28.674 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-07 04:39:28.702 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-07 04:39:28.759 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-07 04:39:28.784 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-07 04:39:28.813 DEBUG [src.node.docker_mananger] No errors found in the waku logs.