172 lines
31 KiB
Plaintext

2026-03-28 04:44:54.266 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-28 04:44:54.267 DEBUG [tests.conftest] Running test: test_auto_shard_relay with id: 2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e
2026-03-28 04:44:54.267 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-28 04:44:54.267 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-28 04:44:54.268 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-28 04:44:54.274 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-28 04:44:54.274 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:54.274 DEBUG [src.node.waku_node] Starting Node...
2026-03-28 04:44:54.275 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-28 04:44:54.276 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-28 04:44:54.276 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.192.12
2026-03-28 04:44:54.276 DEBUG [src.node.docker_mananger] Generated ports ['8095', '8096', '8097', '8098', '8099']
2026-03-28 04:44:54.276 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-28 04:44:54.276 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-28 04:44:54.277 DEBUG [src.node.waku_node] Using volumes []
2026-03-28 04:44:54.277 DEBUG [src.node.docker_mananger] docker run -i -t -p 8095:8095 -p 8096:8096 -p 8097:8097 -p 8098:8098 -p 8099:8099 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=8097 --rest-port=8095 --tcp-port=8096 --discv5-udp-port=8098 --rest-address=0.0.0.0 --nat=extip:172.18.192.12 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=865a216e3cc28a6c23ccfbf1eb8f2a7e1eecccfafa3d65b8fdad93a2b48b3ad0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=8099 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto
2026-03-28 04:44:54.468 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.192.12 waku be87efd87788e8c60a9e9fa05d0cdee9cddbb20aed5176a216a80ae8e7e42723
2026-03-28 04:44:54.503 DEBUG [src.node.docker_mananger] Container started with ID be87efd87788. Setting up logs at ./log/docker/node1_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:54.504 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 8095
2026-03-28 04:44:54.505 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-28 04:44:54.630 ERROR [src.node.docker_mananger] Max retries reached for container de7ab6d41105. Exiting log stream.
2026-03-28 04:44:55.070 ERROR [src.node.docker_mananger] Max retries reached for container ba8fee83494b. Exiting log stream.
2026-03-28 04:44:55.505 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:8095/health" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:55.508 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-28 04:44:55.508 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-28 04:44:55.508 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:8095/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:55.510 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.192.12/tcp/8096/p2p/16Uiu2HAmSUbfFH9iaMHvartaNLjnEeZRB25TvgM8TUwAMtWyV76X","/ip4/172.18.192.12/tcp/8097/ws/p2p/16Uiu2HAmSUbfFH9iaMHvartaNLjnEeZRB25TvgM8TUwAMtWyV76X"],"enrUri":"enr:-L24QMHIdhkbopIAnadZ_lQvAgj3DWss5X-3Xc1xEjE_v4q_JxYW0KCc_7BahNTIDBkY4uW-SLLo3xMXUb9rH-qxfNwCgmlkgnY0gmlwhKwSwAyKbXVsdGlhZGRyc5YACASsEsAMBh-gAAoErBLADAYfod0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzVonG8L9gUntap68ucOwZSj5fEOgRzhmlHERjO_HfNiDdGNwgh-gg3VkcIIfooV3YWt1MgU"}'
2026-03-28 04:44:55.511 INFO [src.node.waku_node] REST service is ready !!
2026-03-28 04:44:55.517 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-28 04:44:55.517 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:55.518 DEBUG [src.node.waku_node] Starting Node...
2026-03-28 04:44:55.518 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-28 04:44:55.519 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-28 04:44:55.519 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.245.180
2026-03-28 04:44:55.519 DEBUG [src.node.docker_mananger] Generated ports ['13780', '13781', '13782', '13783', '13784']
2026-03-28 04:44:55.520 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-28 04:44:55.520 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-28 04:44:55.520 DEBUG [src.node.waku_node] Using volumes []
2026-03-28 04:44:55.520 DEBUG [src.node.docker_mananger] docker run -i -t -p 13780:13780 -p 13781:13781 -p 13782:13782 -p 13783:13783 -p 13784:13784 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=13782 --rest-port=13780 --tcp-port=13781 --discv5-udp-port=13783 --rest-address=0.0.0.0 --nat=extip:172.18.245.180 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=76779a70bbd4e0c5b437542e6c0dbbf3a0118d7078954ff1a564ee9faeaa7f2f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=13784 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMHIdhkbopIAnadZ_lQvAgj3DWss5X-3Xc1xEjE_v4q_JxYW0KCc_7BahNTIDBkY4uW-SLLo3xMXUb9rH-qxfNwCgmlkgnY0gmlwhKwSwAyKbXVsdGlhZGRyc5YACASsEsAMBh-gAAoErBLADAYfod0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzVonG8L9gUntap68ucOwZSj5fEOgRzhmlHERjO_HfNiDdGNwgh-gg3VkcIIfooV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-28 04:44:55.723 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.245.180 waku 6b49acf3668165300e86b73d17c8bd47208ab57d00b667ba6dfccc03f9446b5d
2026-03-28 04:44:55.758 DEBUG [src.node.docker_mananger] Container started with ID 6b49acf36681. Setting up logs at ./log/docker/node2_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:55.759 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 13780
2026-03-28 04:44:55.759 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-28 04:44:56.759 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13780/health" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:56.762 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-28 04:44:56.762 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-28 04:44:56.762 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13780/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:56.765 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.245.180/tcp/13781/p2p/16Uiu2HAmHRySx9XrbYLifJz4ey9w6E4V79aSjhdHMDZ7AXwLDr8D","/ip4/172.18.245.180/tcp/13782/ws/p2p/16Uiu2HAmHRySx9XrbYLifJz4ey9w6E4V79aSjhdHMDZ7AXwLDr8D"],"enrUri":"enr:-L24QG78Y0JftBo3J4nDlM2JT0KQqP7sFOuzfS8A48MTb6qFMZV6utS5_KL9tJ3aDQQDzeyoHBT0i2WpdgPkr9UwV5sCgmlkgnY0gmlwhKwS9bSKbXVsdGlhZGRyc5YACASsEvW0BjXVAAoErBL1tAY11t0DgnJzhQACAQAAiXNlY3AyNTZrMaEDRvTdIQqaGuDKkDCQTuFjk_rPDfVHaxSQgvWaQX3iN1aDdGNwgjXVg3VkcII114V3YWt1MgE"}'
2026-03-28 04:44:56.765 INFO [src.node.waku_node] REST service is ready !!
2026-03-28 04:44:56.766 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13780/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.192.12/tcp/8096/p2p/16Uiu2HAmSUbfFH9iaMHvartaNLjnEeZRB25TvgM8TUwAMtWyV76X"]'
2026-03-28 04:44:56.797 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:44:56.813 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-28 04:44:56.814 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:56.814 DEBUG [src.node.waku_node] Starting Node...
2026-03-28 04:44:56.814 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-28 04:44:56.818 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-28 04:44:56.825 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.204.119
2026-03-28 04:44:56.826 DEBUG [src.node.docker_mananger] Generated ports ['62370', '62371', '62372', '62373', '62374']
2026-03-28 04:44:56.826 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-28 04:44:56.827 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-28 04:44:56.827 DEBUG [src.node.waku_node] Using volumes []
2026-03-28 04:44:56.827 DEBUG [src.node.docker_mananger] docker run -i -t -p 62370:62370 -p 62371:62371 -p 62372:62372 -p 62373:62373 -p 62374:62374 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62372 --rest-port=62370 --tcp-port=62371 --discv5-udp-port=62373 --rest-address=0.0.0.0 --nat=extip:172.18.204.119 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=a78c2faacea91e571a439ff3ac85a9087bb350a01e4bf6cea92f91e34fef60ff --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62374 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMHIdhkbopIAnadZ_lQvAgj3DWss5X-3Xc1xEjE_v4q_JxYW0KCc_7BahNTIDBkY4uW-SLLo3xMXUb9rH-qxfNwCgmlkgnY0gmlwhKwSwAyKbXVsdGlhZGRyc5YACASsEsAMBh-gAAoErBLADAYfod0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzVonG8L9gUntap68ucOwZSj5fEOgRzhmlHERjO_HfNiDdGNwgh-gg3VkcIIfooV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-28 04:44:57.023 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.204.119 waku 3d5a37b376d1fc2e7b3e88967cb86fce6db24527b19e409ef22b93024386c7ad
2026-03-28 04:44:57.055 DEBUG [src.node.docker_mananger] Container started with ID 3d5a37b376d1. Setting up logs at ./log/docker/node3_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:57.056 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 62370
2026-03-28 04:44:57.056 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-28 04:44:58.056 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62370/health" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:58.059 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-28 04:44:58.159 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62370/health" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:58.162 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-28 04:44:58.162 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-28 04:44:58.162 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62370/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:58.165 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.204.119/tcp/62371/p2p/16Uiu2HAmJd4Cgzs52eHJjmKiDhBh68nY53TzxNNBA2uXdnDFpm34","/ip4/172.18.204.119/tcp/62372/ws/p2p/16Uiu2HAmJd4Cgzs52eHJjmKiDhBh68nY53TzxNNBA2uXdnDFpm34"],"enrUri":"enr:-L24QLpLfZ55W6hVFptBavua4kZNO74NZz9tWm0LyYmyQfQgI_y9dRnErgTC2B8kzFJNnXJ94EK7Zd58XeeLLj4AqOkCgmlkgnY0gmlwhKwSzHeKbXVsdGlhZGRyc5YACASsEsx3BvOjAAoErBLMdwbzpN0DgnJzhQACAQAAiXNlY3AyNTZrMaEDWKdQqv5A3bAp7DqYjXPiKOcj5OnsFWQdEtVWI_RpD5-DdGNwgvOjg3VkcILzpYV3YWt1MgE"}'
2026-03-28 04:44:58.165 INFO [src.node.waku_node] REST service is ready !!
2026-03-28 04:44:58.165 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62370/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.192.12/tcp/8096/p2p/16Uiu2HAmSUbfFH9iaMHvartaNLjnEeZRB25TvgM8TUwAMtWyV76X"]'
2026-03-28 04:44:58.185 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:44:58.215 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-28 04:44:58.215 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:58.215 DEBUG [src.node.waku_node] Starting Node...
2026-03-28 04:44:58.216 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-28 04:44:58.219 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-28 04:44:58.220 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.105.28
2026-03-28 04:44:58.220 DEBUG [src.node.docker_mananger] Generated ports ['62322', '62323', '62324', '62325', '62326']
2026-03-28 04:44:58.221 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-28 04:44:58.222 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-28 04:44:58.222 DEBUG [src.node.waku_node] Using volumes []
2026-03-28 04:44:58.223 DEBUG [src.node.docker_mananger] docker run -i -t -p 62322:62322 -p 62323:62323 -p 62324:62324 -p 62325:62325 -p 62326:62326 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62324 --rest-port=62322 --tcp-port=62323 --discv5-udp-port=62325 --rest-address=0.0.0.0 --nat=extip:172.18.105.28 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=76fdee4e6e9aeccef3baf4ec6c40a7b63bb7e489bd9dc51e185187f7b2b7429c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62326 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMHIdhkbopIAnadZ_lQvAgj3DWss5X-3Xc1xEjE_v4q_JxYW0KCc_7BahNTIDBkY4uW-SLLo3xMXUb9rH-qxfNwCgmlkgnY0gmlwhKwSwAyKbXVsdGlhZGRyc5YACASsEsAMBh-gAAoErBLADAYfod0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzVonG8L9gUntap68ucOwZSj5fEOgRzhmlHERjO_HfNiDdGNwgh-gg3VkcIIfooV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-28 04:44:58.441 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.105.28 waku dfc5f2983bbc2cba34080f2a987ca3992b1dd634609dc21d105c222e0a72a232
2026-03-28 04:44:58.475 DEBUG [src.node.docker_mananger] Container started with ID dfc5f2983bbc. Setting up logs at ./log/docker/node4_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:58.476 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 62322
2026-03-28 04:44:58.477 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-28 04:44:59.478 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62322/health" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:59.480 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-28 04:44:59.480 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-28 04:44:59.481 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62322/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:44:59.483 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.105.28/tcp/62323/p2p/16Uiu2HAmFQyftGeXE8pLu5ZRmnHYJVTiUPQHEx4k61P8JWhb4sDn","/ip4/172.18.105.28/tcp/62324/ws/p2p/16Uiu2HAmFQyftGeXE8pLu5ZRmnHYJVTiUPQHEx4k61P8JWhb4sDn"],"enrUri":"enr:-L24QBrRViCDa8rlhn8UowMgWajuMvD0nLdYtB2B1N3GGPqAIWpO8ICb9zogObEyGUaXwZQvvPdrpmtx1ygmFNK-4d0CgmlkgnY0gmlwhKwSaRyKbXVsdGlhZGRyc5YACASsEmkcBvNzAAoErBJpHAbzdN0DgnJzhQACAQAAiXNlY3AyNTZrMaEDKPwpR_CdEEdh2KIzsCPdQQdkTzFopF4aSVsftMCcAkWDdGNwgvNzg3VkcILzdYV3YWt1MgE"}'
2026-03-28 04:44:59.483 INFO [src.node.waku_node] REST service is ready !!
2026-03-28 04:44:59.484 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62322/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.192.12/tcp/8096/p2p/16Uiu2HAmSUbfFH9iaMHvartaNLjnEeZRB25TvgM8TUwAMtWyV76X"]'
2026-03-28 04:44:59.507 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:44:59.521 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-28 04:44:59.523 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:59.524 DEBUG [src.node.waku_node] Starting Node...
2026-03-28 04:44:59.525 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-28 04:44:59.528 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-28 04:44:59.529 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.175.82
2026-03-28 04:44:59.529 DEBUG [src.node.docker_mananger] Generated ports ['15168', '15169', '15170', '15171', '15172']
2026-03-28 04:44:59.529 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-28 04:44:59.529 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-28 04:44:59.529 DEBUG [src.node.waku_node] Using volumes []
2026-03-28 04:44:59.530 DEBUG [src.node.docker_mananger] docker run -i -t -p 15168:15168 -p 15169:15169 -p 15170:15170 -p 15171:15171 -p 15172:15172 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=15170 --rest-port=15168 --tcp-port=15169 --discv5-udp-port=15171 --rest-address=0.0.0.0 --nat=extip:172.18.175.82 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=4caeabcf50e5253b2a5dccb393994f2d5fa7d7d1db19f3afacb44ea2fd9f50de --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=15172 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMHIdhkbopIAnadZ_lQvAgj3DWss5X-3Xc1xEjE_v4q_JxYW0KCc_7BahNTIDBkY4uW-SLLo3xMXUb9rH-qxfNwCgmlkgnY0gmlwhKwSwAyKbXVsdGlhZGRyc5YACASsEsAMBh-gAAoErBLADAYfod0DgnJzhQACAQAAiXNlY3AyNTZrMaEDzVonG8L9gUntap68ucOwZSj5fEOgRzhmlHERjO_HfNiDdGNwgh-gg3VkcIIfooV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-28 04:44:59.739 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.175.82 waku 0ea5ab7e9cd5e024733e3f4c391bea23ef9693d630ddcb6408be72e209fc25b9
2026-03-28 04:44:59.773 DEBUG [src.node.docker_mananger] Container started with ID 0ea5ab7e9cd5. Setting up logs at ./log/docker/node5_2026-03-28_04-44-54__02cb2e3b-2ddf-47ef-b8c9-85e2dd08697e__wakuorg_nwaku:latest.log
2026-03-28 04:44:59.774 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 15168
2026-03-28 04:44:59.774 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-28 04:45:00.774 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15168/health" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.777 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-28 04:45:00.777 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-28 04:45:00.777 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15168/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.780 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.175.82/tcp/15169/p2p/16Uiu2HAmP3e2NcRaAutXSH8m5TN2XCuBApeoqwruPWRmBWLjHBkE","/ip4/172.18.175.82/tcp/15170/ws/p2p/16Uiu2HAmP3e2NcRaAutXSH8m5TN2XCuBApeoqwruPWRmBWLjHBkE"],"enrUri":"enr:-L24QKbM1XjPDffue1bhxhS00vH3va_mDz6ixGlJn8yMimX2W5bFWFoWjIRXa2LdBXhe4tkNTO2q4Hoo1LR4jHljs_kCgmlkgnY0gmlwhKwSr1KKbXVsdGlhZGRyc5YACASsEq9SBjtBAAoErBKvUgY7Qt0DgnJzhQACAQAAiXNlY3AyNTZrMaEDmmI-ugOCvtXNfBezNxQb_qJpE3Llt9CpN5Rr58z9DXODdGNwgjtBg3VkcII7Q4V3YWt1MgE"}'
2026-03-28 04:45:00.780 INFO [src.node.waku_node] REST service is ready !!
2026-03-28 04:45:00.780 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:15168/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.192.12/tcp/8096/p2p/16Uiu2HAmSUbfFH9iaMHvartaNLjnEeZRB25TvgM8TUwAMtWyV76X"]'
2026-03-28 04:45:00.805 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.806 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:8095/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-28 04:45:00.821 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.822 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13780/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-28 04:45:00.826 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.827 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62370/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-28 04:45:00.834 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.835 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62322/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-28 04:45:00.839 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.839 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:15168/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-28 04:45:00.843 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.843 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:8095/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-28 04:45:00.858 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-28 04:45:00.859 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-28 04:45:00.959 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-28 04:45:00.960 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:8095/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.962 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774673100843745512,"ephemeral":false,"proof":""}]'
2026-03-28 04:45:00.964 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-28 04:45:00.964 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13780/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.967 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774673100843745512,"ephemeral":false,"proof":""}]'
2026-03-28 04:45:00.968 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-28 04:45:00.968 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62370/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.970 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774673100843745512,"ephemeral":false,"proof":""}]'
2026-03-28 04:45:00.972 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-28 04:45:00.972 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62322/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.975 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774673100843745512,"ephemeral":false,"proof":""}]'
2026-03-28 04:45:00.976 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-28 04:45:00.976 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15168/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-28 04:45:00.979 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774673100843745512,"ephemeral":false,"proof":""}]'
2026-03-28 04:45:00.981 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-28 04:45:00.983 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-28 04:45:00.983 DEBUG [src.node.waku_node] Stopping container with id be87efd87788
2026-03-28 04:45:01.509 DEBUG [src.node.waku_node] Container stopped.
2026-03-28 04:45:01.511 DEBUG [src.node.waku_node] Stopping container with id 6b49acf36681
2026-03-28 04:45:02.048 DEBUG [src.node.waku_node] Container stopped.
2026-03-28 04:45:02.049 DEBUG [src.node.waku_node] Stopping container with id 3d5a37b376d1
2026-03-28 04:45:02.323 ERROR [src.node.docker_mananger] Max retries reached for container be87efd87788. Exiting log stream.
2026-03-28 04:45:02.537 DEBUG [src.node.waku_node] Container stopped.
2026-03-28 04:45:02.538 DEBUG [src.node.waku_node] Stopping container with id dfc5f2983bbc
2026-03-28 04:45:02.862 ERROR [src.node.docker_mananger] Max retries reached for container 6b49acf36681. Exiting log stream.
2026-03-28 04:45:02.994 DEBUG [src.node.waku_node] Container stopped.
2026-03-28 04:45:02.994 DEBUG [src.node.waku_node] Stopping container with id 0ea5ab7e9cd5
2026-03-28 04:45:03.351 ERROR [src.node.docker_mananger] Max retries reached for container 3d5a37b376d1. Exiting log stream.
2026-03-28 04:45:03.435 DEBUG [src.node.waku_node] Container stopped.
2026-03-28 04:45:03.438 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-28 04:45:03.475 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-28 04:45:03.494 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-28 04:45:03.513 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-28 04:45:03.530 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-28 04:45:03.545 DEBUG [src.node.docker_mananger] No errors found in the waku logs.