192 lines
35 KiB
Plaintext

2026-03-08 04:33:56.329 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-08 04:33:56.329 DEBUG [tests.conftest] Running test: test_last_node_to_start_publishes with id: 2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f
2026-03-08 04:33:56.330 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-08 04:33:56.330 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-08 04:33:56.330 DEBUG [src.steps.relay] Running fixture setup: setup_main_relay_nodes
2026-03-08 04:33:56.339 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:56.339 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:33:56.339 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:56.339 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:56.341 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:56.341 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.209.69
2026-03-08 04:33:56.342 DEBUG [src.node.docker_mananger] Generated ports ['50422', '50423', '50424', '50425', '50426']
2026-03-08 04:33:56.342 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:56.342 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:56.342 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:56.342 DEBUG [src.node.docker_mananger] docker run -i -t -p 50422:50422 -p 50423:50423 -p 50424:50424 -p 50425:50425 -p 50426:50426 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=50424 --rest-port=50422 --tcp-port=50423 --discv5-udp-port=50425 --rest-address=0.0.0.0 --nat=extip:172.18.209.69 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=67cfe5c3506b46cb9c5bd1a2a37b4f0eb5339ef9ae7c75adfb4ecb1d2785d912 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=50426 --metrics-logging=true --relay=true
2026-03-08 04:33:56.475 ERROR [src.node.docker_mananger] Max retries reached for container 1aea140d521e. Exiting log stream.
2026-03-08 04:33:56.560 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.209.69 waku d20a3ba1c4b0b6558855fb261e258c129ef6b4dd7fe81e0624150ae4eec5aca6
2026-03-08 04:33:56.598 DEBUG [src.node.docker_mananger] Container started with ID d20a3ba1c4b0. Setting up logs at ./log/docker/node1_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:33:56.599 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 50422
2026-03-08 04:33:56.600 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:33:57.127 ERROR [src.node.docker_mananger] Max retries reached for container 2f6c3305596e. Exiting log stream.
2026-03-08 04:33:57.600 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50422/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:57.603 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:33:57.604 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:33:57.604 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50422/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:57.606 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.209.69/tcp/50423/p2p/16Uiu2HAmAshxXfw9yNh94nw9Z7NSAXFzaZ84ZzEemBrGqkHEFrRB","/ip4/172.18.209.69/tcp/50424/ws/p2p/16Uiu2HAmAshxXfw9yNh94nw9Z7NSAXFzaZ84ZzEemBrGqkHEFrRB"],"enrUri":"enr:-L24QNDLfS8jb5bilBu0yqCpdy1zh6KqySKyCItWqMYEOf-CUp8rjhi9CHivLrT6Uc94howhOAdMQQ1t7U-gNs6Cnu8CgmlkgnY0gmlwhKwS0UWKbXVsdGlhZGRyc5YACASsEtFFBsT3AAoErBLRRQbE-N0DgnJzhQADAQAAiXNlY3AyNTZrMaEC5YqkRmUAvxUnz89sbDE_iW9RVYHuR3bA-09AQZhSgR6DdGNwgsT3g3VkcILE-YV3YWt1MgE"}'
2026-03-08 04:33:57.606 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:33:57.614 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:57.614 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:33:57.614 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:57.614 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:57.616 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:57.616 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.110.75
2026-03-08 04:33:57.616 DEBUG [src.node.docker_mananger] Generated ports ['57247', '57248', '57249', '57250', '57251']
2026-03-08 04:33:57.616 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:57.617 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:57.617 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:57.617 DEBUG [src.node.docker_mananger] docker run -i -t -p 57247:57247 -p 57248:57248 -p 57249:57249 -p 57250:57250 -p 57251:57251 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=57249 --rest-port=57247 --tcp-port=57248 --discv5-udp-port=57250 --rest-address=0.0.0.0 --nat=extip:172.18.110.75 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=c0af7feddadad9a83eebdce5ae7f46392a483dda3a3bc8dbd5ef9ab8882f6ea8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=57251 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QNDLfS8jb5bilBu0yqCpdy1zh6KqySKyCItWqMYEOf-CUp8rjhi9CHivLrT6Uc94howhOAdMQQ1t7U-gNs6Cnu8CgmlkgnY0gmlwhKwS0UWKbXVsdGlhZGRyc5YACASsEtFFBsT3AAoErBLRRQbE-N0DgnJzhQADAQAAiXNlY3AyNTZrMaEC5YqkRmUAvxUnz89sbDE_iW9RVYHuR3bA-09AQZhSgR6DdGNwgsT3g3VkcILE-YV3YWt1MgE
2026-03-08 04:33:57.812 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.110.75 waku d939195bd47c17d374e3ae31648c4e57f909b5a9eaa2e54b75b3b1fc85e9a4c6
2026-03-08 04:33:57.850 DEBUG [src.node.docker_mananger] Container started with ID d939195bd47c. Setting up logs at ./log/docker/node2_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:33:57.851 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 57247
2026-03-08 04:33:57.851 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:33:58.851 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57247/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:58.862 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:33:58.863 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:33:58.863 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57247/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:33:58.872 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.110.75/tcp/57248/p2p/16Uiu2HAm8pFy9AHBUq4EhZibXs7fP2H6b6q72RN2uYDjgSZGVhms","/ip4/172.18.110.75/tcp/57249/ws/p2p/16Uiu2HAm8pFy9AHBUq4EhZibXs7fP2H6b6q72RN2uYDjgSZGVhms"],"enrUri":"enr:-L24QIMgDXx4KZbFMKiShbd6dW63jlHkXQ8E_jJZ4lpFV8Iefn3Eb7rIB9HpKYgbs3JcTpKykTd9bR9DrfKvvuCqR04CgmlkgnY0gmlwhKwSbkuKbXVsdGlhZGRyc5YACASsEm5LBt-gAAoErBJuSwbfod0DgnJzhQADAQAAiXNlY3AyNTZrMaECxvEjmQlmr10Cdca7xgRCEIvSje4KG4234w8vq0riz9qDdGNwgt-gg3VkcILfooV3YWt1MgE"}'
2026-03-08 04:33:58.872 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:33:58.873 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:57247/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.209.69/tcp/50423/p2p/16Uiu2HAmAshxXfw9yNh94nw9Z7NSAXFzaZ84ZzEemBrGqkHEFrRB"]'
2026-03-08 04:33:58.877 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:33:58.877 DEBUG [src.steps.relay] Running fixture setup: setup_optional_relay_nodes
2026-03-08 04:33:58.884 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:33:58.885 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:33:58.885 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:33:58.885 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:33:58.887 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:33:58.887 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.222.161
2026-03-08 04:33:58.887 DEBUG [src.node.docker_mananger] Generated ports ['30301', '30302', '30303', '30304', '30305']
2026-03-08 04:33:58.887 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:33:58.887 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:33:58.887 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:33:58.888 DEBUG [src.node.docker_mananger] docker run -i -t -p 30301:30301 -p 30302:30302 -p 30303:30303 -p 30304:30304 -p 30305:30305 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=30303 --rest-port=30301 --tcp-port=30302 --discv5-udp-port=30304 --rest-address=0.0.0.0 --nat=extip:172.18.222.161 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=9db6a4d425280a5ebdb434dec48d728de9dbc3e972d91e8a7f650d091ee8aed9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=30305 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QNDLfS8jb5bilBu0yqCpdy1zh6KqySKyCItWqMYEOf-CUp8rjhi9CHivLrT6Uc94howhOAdMQQ1t7U-gNs6Cnu8CgmlkgnY0gmlwhKwS0UWKbXVsdGlhZGRyc5YACASsEtFFBsT3AAoErBLRRQbE-N0DgnJzhQADAQAAiXNlY3AyNTZrMaEC5YqkRmUAvxUnz89sbDE_iW9RVYHuR3bA-09AQZhSgR6DdGNwgsT3g3VkcILE-YV3YWt1MgE
2026-03-08 04:33:59.104 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.222.161 waku 287fc3cce7e56348ccb50c894a4c78b384165f4a4d06a5382fa766e9f3e6888f
2026-03-08 04:33:59.144 DEBUG [src.node.docker_mananger] Container started with ID 287fc3cce7e5. Setting up logs at ./log/docker/node3_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:33:59.145 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 30301
2026-03-08 04:33:59.145 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:34:00.146 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30301/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:00.166 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:34:00.168 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:34:00.169 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30301/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:00.173 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.222.161/tcp/30302/p2p/16Uiu2HAmT7ZJBPxYi2ZNrVeEcM1iYH3xefZGJgGFbiNm2R67mTV8","/ip4/172.18.222.161/tcp/30303/ws/p2p/16Uiu2HAmT7ZJBPxYi2ZNrVeEcM1iYH3xefZGJgGFbiNm2R67mTV8"],"enrUri":"enr:-L24QLCXyl0764IglOvQisP88AwIUge2ZhEYN72gEozs5G0DX8A77fko3sU3VuSC73Rjq0eZZYN0l2CyJzhtQE_0AGICgmlkgnY0gmlwhKwS3qGKbXVsdGlhZGRyc5YACASsEt6hBnZeAAoErBLeoQZ2X90DgnJzhQADAQAAiXNlY3AyNTZrMaED1tH4eR_vn9iEJdOLh2L8XI0Hx7Dh4fHnbu_F3r4Dd-eDdGNwgnZeg3VkcIJ2YIV3YWt1MgE"}'
2026-03-08 04:34:00.173 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:34:00.174 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30301/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.209.69/tcp/50423/p2p/16Uiu2HAmAshxXfw9yNh94nw9Z7NSAXFzaZ84ZzEemBrGqkHEFrRB"]'
2026-03-08 04:34:00.177 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:00.185 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:34:00.185 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:34:00.185 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:34:00.186 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:34:00.187 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:34:00.187 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.195.159
2026-03-08 04:34:00.188 DEBUG [src.node.docker_mananger] Generated ports ['58577', '58578', '58579', '58580', '58581']
2026-03-08 04:34:00.188 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:34:00.188 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:34:00.188 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:34:00.188 DEBUG [src.node.docker_mananger] docker run -i -t -p 58577:58577 -p 58578:58578 -p 58579:58579 -p 58580:58580 -p 58581:58581 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=58579 --rest-port=58577 --tcp-port=58578 --discv5-udp-port=58580 --rest-address=0.0.0.0 --nat=extip:172.18.195.159 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3e3eabdc3f3f00af4b7febc0c952cebed1a8d5fde10d667ed8d2fafc8e25c8ad --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=58581 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QNDLfS8jb5bilBu0yqCpdy1zh6KqySKyCItWqMYEOf-CUp8rjhi9CHivLrT6Uc94howhOAdMQQ1t7U-gNs6Cnu8CgmlkgnY0gmlwhKwS0UWKbXVsdGlhZGRyc5YACASsEtFFBsT3AAoErBLRRQbE-N0DgnJzhQADAQAAiXNlY3AyNTZrMaEC5YqkRmUAvxUnz89sbDE_iW9RVYHuR3bA-09AQZhSgR6DdGNwgsT3g3VkcILE-YV3YWt1MgE
2026-03-08 04:34:00.410 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.195.159 waku 358003d4d9f78e53b428f3f9073ab69eafa39071d94492648f23bc0402b51b84
2026-03-08 04:34:00.455 DEBUG [src.node.docker_mananger] Container started with ID 358003d4d9f7. Setting up logs at ./log/docker/node4_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:34:00.455 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 58577
2026-03-08 04:34:00.455 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:34:01.456 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58577/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:01.460 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:34:01.461 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:34:01.461 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58577/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:01.470 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.195.159/tcp/58578/p2p/16Uiu2HAm76DNB5kqV4mxnKMbcS3dW3apBVceshPWh7WpVvNv2kJc","/ip4/172.18.195.159/tcp/58579/ws/p2p/16Uiu2HAm76DNB5kqV4mxnKMbcS3dW3apBVceshPWh7WpVvNv2kJc"],"enrUri":"enr:-L24QBSxmpHbuAsG-134pOBn__o0O9G1Z1R245gyerzsgNWBCIFfOOQFJ7lOFrRspSPyqYTFFLR6CwbxYo-kPzbCgNICgmlkgnY0gmlwhKwSw5-KbXVsdGlhZGRyc5YACASsEsOfBuTSAAoErBLDnwbk090DgnJzhQADAQAAiXNlY3AyNTZrMaECrVAddNbaOj3_KPfbuky1fUgflslNlvz_Ttl0tv2VB-GDdGNwguTSg3VkcILk1IV3YWt1MgE"}'
2026-03-08 04:34:01.471 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:34:01.471 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58577/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.209.69/tcp/50423/p2p/16Uiu2HAmAshxXfw9yNh94nw9Z7NSAXFzaZ84ZzEemBrGqkHEFrRB"]'
2026-03-08 04:34:01.475 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:01.483 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-08 04:34:01.483 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:34:01.483 DEBUG [src.node.waku_node] Starting Node...
2026-03-08 04:34:01.483 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-08 04:34:01.485 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-08 04:34:01.485 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.142.195
2026-03-08 04:34:01.485 DEBUG [src.node.docker_mananger] Generated ports ['35374', '35375', '35376', '35377', '35378']
2026-03-08 04:34:01.485 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-08 04:34:01.485 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-08 04:34:01.486 DEBUG [src.node.waku_node] Using volumes []
2026-03-08 04:34:01.486 DEBUG [src.node.docker_mananger] docker run -i -t -p 35374:35374 -p 35375:35375 -p 35376:35376 -p 35377:35377 -p 35378:35378 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=35376 --rest-port=35374 --tcp-port=35375 --discv5-udp-port=35377 --rest-address=0.0.0.0 --nat=extip:172.18.142.195 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=885df586fe83f3b4cde31e9923fce7a2dd1acc84affe59e9fd07ad6b0fb6681f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=35378 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QNDLfS8jb5bilBu0yqCpdy1zh6KqySKyCItWqMYEOf-CUp8rjhi9CHivLrT6Uc94howhOAdMQQ1t7U-gNs6Cnu8CgmlkgnY0gmlwhKwS0UWKbXVsdGlhZGRyc5YACASsEtFFBsT3AAoErBLRRQbE-N0DgnJzhQADAQAAiXNlY3AyNTZrMaEC5YqkRmUAvxUnz89sbDE_iW9RVYHuR3bA-09AQZhSgR6DdGNwgsT3g3VkcILE-YV3YWt1MgE
2026-03-08 04:34:01.703 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.142.195 waku 6f974c9e538e79559ed74f608b7fe950f593ddc3c3fe6e307e3f83ea4652d655
2026-03-08 04:34:01.744 DEBUG [src.node.docker_mananger] Container started with ID 6f974c9e538e. Setting up logs at ./log/docker/node5_2026-03-08_04-33-56__a82b035e-fc06-4739-9946-e004073b9b3f__wakuorg_nwaku:latest.log
2026-03-08 04:34:01.744 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 35374
2026-03-08 04:34:01.745 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-08 04:34:02.745 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35374/health" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.757 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-08 04:34:02.760 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-08 04:34:02.762 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35374/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.768 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.142.195/tcp/35375/p2p/16Uiu2HAm5BvsVxGXPmrqo6J1jwDzUUSMQ2WVn6yJTjeNyj6UFdYQ","/ip4/172.18.142.195/tcp/35376/ws/p2p/16Uiu2HAm5BvsVxGXPmrqo6J1jwDzUUSMQ2WVn6yJTjeNyj6UFdYQ"],"enrUri":"enr:-L24QChL-peet-NDL6JqdCSg-NRcQcdO-H_lXHqTsglVnBTJAdvu3mxzeVTlyU2FNycBDE7BbLPKo7G6fDpVjlnkin0CgmlkgnY0gmlwhKwSjsOKbXVsdGlhZGRyc5YACASsEo7DBoovAAoErBKOwwaKMN0DgnJzhQADAQAAiXNlY3AyNTZrMaECkQ-TZ5Oi46I2f2FE6_eXXb78_oYFqpWhOkOYawqaDK2DdGNwgoovg3VkcIKKMYV3YWt1MgE"}'
2026-03-08 04:34:02.769 INFO [src.node.waku_node] REST service is ready !!
2026-03-08 04:34:02.770 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35374/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.209.69/tcp/50423/p2p/16Uiu2HAmAshxXfw9yNh94nw9Z7NSAXFzaZ84ZzEemBrGqkHEFrRB"]'
2026-03-08 04:34:02.774 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.775 DEBUG [src.steps.relay] Running fixture setup: subscribe_main_relay_nodes
2026-03-08 04:34:02.775 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:50422/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:34:02.786 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.787 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:57247/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:34:02.795 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.796 DEBUG [src.steps.relay] Running fixture setup: subscribe_optional_relay_nodes
2026-03-08 04:34:02.797 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30301/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:34:02.805 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.806 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58577/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:34:02.814 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.816 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35374/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-08 04:34:02.824 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.826 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:50422/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:34:02.840 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.841 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:34:02.941 DEBUG [src.steps.relay] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:02.942 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50422/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.945 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442825965568,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:02.947 DEBUG [src.steps.relay] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:02.947 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57247/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.950 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442825965568,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:02.952 DEBUG [src.steps.relay] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:02.952 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30301/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.955 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442825965568,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:02.957 DEBUG [src.steps.relay] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:02.957 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58577/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.960 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442825965568,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:02.962 DEBUG [src.steps.relay] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:02.962 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35374/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:02.965 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442825965568,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:02.966 INFO [src.steps.relay] WARM UP successful!!
2026-03-08 04:34:02.969 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35374/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-08 04:34:02.975 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-08 04:34:02.976 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-08 04:34:03.077 DEBUG [src.steps.relay] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.077 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50422/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.080 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442969629957,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.082 DEBUG [src.steps.relay] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.082 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57247/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.085 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442969629957,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.087 DEBUG [src.steps.relay] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.087 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30301/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.089 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442969629957,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.091 DEBUG [src.steps.relay] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.091 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58577/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.094 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442969629957,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.096 DEBUG [src.steps.relay] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-08 04:34:03.096 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35374/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-08 04:34:03.099 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772944442969629957,"ephemeral":false,"proof":""}]'
2026-03-08 04:34:03.103 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-08 04:34:03.104 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-08 04:34:03.104 DEBUG [src.node.waku_node] Stopping container with id d20a3ba1c4b0
2026-03-08 04:34:03.777 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:03.778 DEBUG [src.node.waku_node] Stopping container with id d939195bd47c
2026-03-08 04:34:04.398 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:04.400 DEBUG [src.node.waku_node] Stopping container with id 287fc3cce7e5
2026-03-08 04:34:04.591 ERROR [src.node.docker_mananger] Max retries reached for container d20a3ba1c4b0. Exiting log stream.
2026-03-08 04:34:05.009 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:05.010 DEBUG [src.node.waku_node] Stopping container with id 358003d4d9f7
2026-03-08 04:34:05.217 ERROR [src.node.docker_mananger] Max retries reached for container d939195bd47c. Exiting log stream.
2026-03-08 04:34:05.671 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:05.673 DEBUG [src.node.waku_node] Stopping container with id 6f974c9e538e
2026-03-08 04:34:05.827 ERROR [src.node.docker_mananger] Max retries reached for container 287fc3cce7e5. Exiting log stream.
2026-03-08 04:34:06.289 DEBUG [src.node.waku_node] Container stopped.
2026-03-08 04:34:06.293 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-08 04:34:06.308 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:06.314 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:06.319 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:06.325 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-08 04:34:06.330 DEBUG [src.node.docker_mananger] No errors found in the waku logs.