2026-02-25 15:39:14 +00:00

188 lines
34 KiB
Plaintext

2026-02-25 15:25:51.415 DEBUG [tests.conftest] Running fixture setup: test_id
2026-02-25 15:25:51.415 DEBUG [tests.conftest] Running test: test_multiple_receiving_nodes with id: 2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a
2026-02-25 15:25:51.416 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-02-25 15:25:51.416 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-02-25 15:25:51.423 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 15:25:51.424 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node1_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:51.424 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 15:25:51.424 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 15:25:51.425 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 15:25:51.425 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.161.115
2026-02-25 15:25:51.426 DEBUG [src.node.docker_mananger] Generated ports ['33593', '33594', '33595', '33596', '33597']
2026-02-25 15:25:51.426 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 15:25:51.426 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 15:25:51.426 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 15:25:51.426 DEBUG [src.node.docker_mananger] docker run -i -t -p 33593:33593 -p 33594:33594 -p 33595:33595 -p 33596:33596 -p 33597:33597 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33595 --rest-port=33593 --tcp-port=33594 --discv5-udp-port=33596 --rest-address=0.0.0.0 --nat=extip:172.18.161.115 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ecce7a4f04e0effa60853e41417cbdbacba6c46830b5b31dc99d16b7fcdefd1f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33597 --metrics-logging=true --lightpush=true --relay=true
2026-02-25 15:25:51.617 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.161.115 waku dddd3052876a9c8691935ceccfd5819508d9ebe2633842c94d2c251f4594fc27
2026-02-25 15:25:51.649 DEBUG [src.node.docker_mananger] Container started with ID dddd3052876a. Setting up logs at ./log/docker/receiving_node1_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:51.649 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33593
2026-02-25 15:25:51.650 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 15:25:51.667 ERROR [src.node.docker_mananger] Max retries reached for container 2f33c250c906. Exiting log stream.
2026-02-25 15:25:52.176 ERROR [src.node.docker_mananger] Max retries reached for container 58f0ad1ef250. Exiting log stream.
2026-02-25 15:25:52.650 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33593/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:52.653 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 15:25:52.653 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 15:25:52.654 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33593/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:52.656 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.161.115/tcp/33594/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1","/ip4/172.18.161.115/tcp/33595/ws/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1"],"enrUri":"enr:-L24QCou42N3q8jcL5-Ztq0moaoHZ45iA1jKc3UuKz8yVPqoE4MN8uY12YXORzLTyjHCjbODswzbUzpc587BsLm88ycCgmlkgnY0gmlwhKwSoXOKbXVsdGlhZGRyc5YACASsEqFzBoM6AAoErBKhcwaDO90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrngfcs4nN8cd4O-8DowrU42RvWsltNtVKcbjRZqKbyCDdGNwgoM6g3VkcIKDPIV3YWt1Mgk"}'
2026-02-25 15:25:52.656 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 15:25:52.663 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 15:25:52.663 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node2_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:52.663 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 15:25:52.664 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 15:25:52.665 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 15:25:52.665 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.214.59
2026-02-25 15:25:52.665 DEBUG [src.node.docker_mananger] Generated ports ['55176', '55177', '55178', '55179', '55180']
2026-02-25 15:25:52.665 DEBUG [src.node.waku_node] Cluster id was resolved to: 3
2026-02-25 15:25:52.666 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-25 15:25:52.666 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 15:25:52.666 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 15:25:52.666 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 15:25:52.666 DEBUG [src.node.docker_mananger] docker run -i -t -p 55176:55176 -p 55177:55177 -p 55178:55178 -p 55179:55179 -p 55180:55180 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=55178 --rest-port=55176 --tcp-port=55177 --discv5-udp-port=55179 --rest-address=0.0.0.0 --nat=extip:172.18.214.59 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=cbff71fc08b0ec4f29e57fd387d6dae8e231bbd3ae2d3e9dd1abef65d2b11269 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=55180 --metrics-logging=true --lightpush=true --relay=true
2026-02-25 15:25:52.854 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.214.59 waku 17624d8a4f4c1fea9a8d75aebbecbf9d17c83c9e6a8e1e266a32131c30c02a92
2026-02-25 15:25:52.885 DEBUG [src.node.docker_mananger] Container started with ID 17624d8a4f4c. Setting up logs at ./log/docker/receiving_node2_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:52.885 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 55176
2026-02-25 15:25:52.885 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 15:25:53.887 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55176/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:53.889 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 15:25:53.890 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 15:25:53.890 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55176/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:53.892 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.214.59/tcp/55177/p2p/16Uiu2HAmGZugaf6yUZY2LRQZQYLEAm5y14hC3aRQvnTt2soRPKrM","/ip4/172.18.214.59/tcp/55178/ws/p2p/16Uiu2HAmGZugaf6yUZY2LRQZQYLEAm5y14hC3aRQvnTt2soRPKrM"],"enrUri":"enr:-L24QH5eQQHzSwJhLMl9iTeHFiNUZglsBfFQUvJP7AgkOW7PDCqT3PiyDjCuSqBaPakXuW3qp3XNYo5h9kuv2L_cOuYCgmlkgnY0gmlwhKwS1juKbXVsdGlhZGRyc5YACASsEtY7BteJAAoErBLWOwbXit0DgnJzhQADAQAAiXNlY3AyNTZrMaEDOiGQ8_SOSmPyZAyS_2ZEVIdry4HbqT-f8q0JPI0ruWaDdGNwgteJg3VkcILXi4V3YWt1Mgk"}'
2026-02-25 15:25:53.892 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 15:25:53.893 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:55176/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.161.115/tcp/33594/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1"]'
2026-02-25 15:25:53.928 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:53.938 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 15:25:53.938 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node3_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:53.938 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 15:25:53.938 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 15:25:53.940 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 15:25:53.940 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.64.84
2026-02-25 15:25:53.940 DEBUG [src.node.docker_mananger] Generated ports ['27694', '27695', '27696', '27697', '27698']
2026-02-25 15:25:53.940 DEBUG [src.node.waku_node] Cluster id was resolved to: 3
2026-02-25 15:25:53.940 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-25 15:25:53.941 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 15:25:53.941 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 15:25:53.941 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 15:25:53.941 DEBUG [src.node.docker_mananger] docker run -i -t -p 27694:27694 -p 27695:27695 -p 27696:27696 -p 27697:27697 -p 27698:27698 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=27696 --rest-port=27694 --tcp-port=27695 --discv5-udp-port=27697 --rest-address=0.0.0.0 --nat=extip:172.18.64.84 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3ccb1bf974ddaef8533adcffb4ba1b4b015fcbe8062bda99e4b35dea4add875b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=27698 --metrics-logging=true --lightpush=true --relay=true
2026-02-25 15:25:54.136 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.64.84 waku 77b5607c70fd582bb299652283809f4c652fc143c7f9a8880ce4eefc85a0a4ee
2026-02-25 15:25:54.174 DEBUG [src.node.docker_mananger] Container started with ID 77b5607c70fd. Setting up logs at ./log/docker/receiving_node3_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:54.174 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 27694
2026-02-25 15:25:54.174 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 15:25:55.175 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27694/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:55.177 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 15:25:55.178 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 15:25:55.178 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27694/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:55.180 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.64.84/tcp/27695/p2p/16Uiu2HAkwuNPAxWZNCRWxtsJwx7BHNLRNArE2oiXr5bYWptzCW5S","/ip4/172.18.64.84/tcp/27696/ws/p2p/16Uiu2HAkwuNPAxWZNCRWxtsJwx7BHNLRNArE2oiXr5bYWptzCW5S"],"enrUri":"enr:-L24QJmgB1NjfVH09uRFMCNH6--2bAHrSU5iyFjv_AHhOiYcY8CCWySF9cSX9Tt2UPScNgKHj8Ucr2-4zRvtf7Vba6MCgmlkgnY0gmlwhKwSQFSKbXVsdGlhZGRyc5YACASsEkBUBmwvAAoErBJAVAZsMN0DgnJzhQADAQAAiXNlY3AyNTZrMaECJM-9Vz1N25YFkS6YvqPwv_v9CYE6P4z59w7UmWl7Nn2DdGNwgmwvg3VkcIJsMYV3YWt1Mgk"}'
2026-02-25 15:25:55.180 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 15:25:55.181 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27694/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.161.115/tcp/33594/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1"]'
2026-02-25 15:25:55.218 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:55.219 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27694/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.214.59/tcp/55177/p2p/16Uiu2HAmGZugaf6yUZY2LRQZQYLEAm5y14hC3aRQvnTt2soRPKrM"]'
2026-02-25 15:25:55.244 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:55.257 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 15:25:55.257 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node4_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:55.259 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 15:25:55.259 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 15:25:55.264 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 15:25:55.265 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.71.162
2026-02-25 15:25:55.265 DEBUG [src.node.docker_mananger] Generated ports ['51945', '51946', '51947', '51948', '51949']
2026-02-25 15:25:55.265 DEBUG [src.node.waku_node] Cluster id was resolved to: 3
2026-02-25 15:25:55.265 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-02-25 15:25:55.266 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 15:25:55.267 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 15:25:55.268 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 15:25:55.269 DEBUG [src.node.docker_mananger] docker run -i -t -p 51945:51945 -p 51946:51946 -p 51947:51947 -p 51948:51948 -p 51949:51949 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51947 --rest-port=51945 --tcp-port=51946 --discv5-udp-port=51948 --rest-address=0.0.0.0 --nat=extip:172.18.71.162 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=cbdbf4a03180ffa0092dafc6d92ea00fd2a69f2ab538fe54718da78edfdaece0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51949 --metrics-logging=true --lightpush=true --relay=true
2026-02-25 15:25:55.470 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.71.162 waku c4aa1ece260a117dbf7de856260c9155768bcac13639d2c73dd8f0048f04e4de
2026-02-25 15:25:55.507 DEBUG [src.node.docker_mananger] Container started with ID c4aa1ece260a. Setting up logs at ./log/docker/receiving_node4_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:55.508 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51945
2026-02-25 15:25:55.508 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 15:25:56.509 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51945/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:56.511 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 15:25:56.512 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 15:25:56.512 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51945/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:56.514 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.71.162/tcp/51946/p2p/16Uiu2HAmVnVQy9WX6RaY7Pq5hFEwAaSfDtA1gZjKvP84xpDGJtWh","/ip4/172.18.71.162/tcp/51947/ws/p2p/16Uiu2HAmVnVQy9WX6RaY7Pq5hFEwAaSfDtA1gZjKvP84xpDGJtWh"],"enrUri":"enr:-L24QK4YkP2-mqT0Apk7u_-Ih4l3RWZj6LBB8UlLf5LdVoDRRNf2RD_d2rKapIpD0VykNpwKfKzrTTPfGklIVbS1e2QCgmlkgnY0gmlwhKwSR6KKbXVsdGlhZGRyc5YACASsEkeiBsrqAAoErBJHogbK690DgnJzhQADAQAAiXNlY3AyNTZrMaED_oKbBIFsQg9RXKZPlY_YL5P6ZMTL0VhE9oxqbVS5CT6DdGNwgsrqg3VkcILK7IV3YWt1Mgk"}'
2026-02-25 15:25:56.514 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 15:25:56.515 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51945/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.161.115/tcp/33594/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1"]'
2026-02-25 15:25:56.559 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:56.560 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51945/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.214.59/tcp/55177/p2p/16Uiu2HAmGZugaf6yUZY2LRQZQYLEAm5y14hC3aRQvnTt2soRPKrM"]'
2026-02-25 15:25:56.585 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:56.588 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51945/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.64.84/tcp/27695/p2p/16Uiu2HAkwuNPAxWZNCRWxtsJwx7BHNLRNArE2oiXr5bYWptzCW5S"]'
2026-02-25 15:25:56.616 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:56.628 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-25 15:25:56.629 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/lightpush_node1_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:56.629 DEBUG [src.node.waku_node] Starting Node...
2026-02-25 15:25:56.629 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-25 15:25:56.631 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-25 15:25:56.632 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.156.112
2026-02-25 15:25:56.632 DEBUG [src.node.docker_mananger] Generated ports ['63183', '63184', '63185', '63186', '63187']
2026-02-25 15:25:56.632 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-25 15:25:56.632 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-25 15:25:56.633 DEBUG [src.node.waku_node] Using volumes []
2026-02-25 15:25:56.633 DEBUG [src.node.docker_mananger] docker run -i -t -p 63183:63183 -p 63184:63184 -p 63185:63185 -p 63186:63186 -p 63187:63187 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=63185 --rest-port=63183 --tcp-port=63184 --discv5-udp-port=63186 --rest-address=0.0.0.0 --nat=extip:172.18.156.112 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3d4bdeafefccbbcfbfe13bcaa91ea7beb4aa54a5a642db37ba8db1b1e0abe74f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=63187 --metrics-logging=true --discv5-bootstrap-node=enr:-L24QCou42N3q8jcL5-Ztq0moaoHZ45iA1jKc3UuKz8yVPqoE4MN8uY12YXORzLTyjHCjbODswzbUzpc587BsLm88ycCgmlkgnY0gmlwhKwSoXOKbXVsdGlhZGRyc5YACASsEqFzBoM6AAoErBKhcwaDO90DgnJzhQADAQAAiXNlY3AyNTZrMaEDrngfcs4nN8cd4O-8DowrU42RvWsltNtVKcbjRZqKbyCDdGNwgoM6g3VkcIKDPIV3YWt1Mgk --lightpushnode=/ip4/172.18.161.115/tcp/33594/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1 --lightpush=true --relay=true
2026-02-25 15:25:56.830 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.156.112 waku 2ba8a24aa6912301e53535df082b28d1b032f13211dbe0b48c59361dfa5fde34
2026-02-25 15:25:56.888 DEBUG [src.node.docker_mananger] Container started with ID 2ba8a24aa691. Setting up logs at ./log/docker/lightpush_node1_2026-02-25_15-25-51__ff0572f2-58d5-49aa-8141-f5a93373cc3a__wakuorg_nwaku:latest.log
2026-02-25 15:25:56.889 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 63183
2026-02-25 15:25:56.889 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-25 15:25:57.889 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63183/health" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:57.911 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-25 15:25:57.914 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-25 15:25:57.914 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63183/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:57.919 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.156.112/tcp/63184/p2p/16Uiu2HAm59YfYmb6vaYKV4sT5BKcQTKQNvJ7QBW7EtLWoUUgC4zP","/ip4/172.18.156.112/tcp/63185/ws/p2p/16Uiu2HAm59YfYmb6vaYKV4sT5BKcQTKQNvJ7QBW7EtLWoUUgC4zP"],"enrUri":"enr:-L24QBKedC8AMZFXyeQn8YYSWtFRkvLpMi74xyjeX47jfxdHPdvvOiDn7n2dv4Np5YEIQ00sFPQdNosaENM7q5_lpA0CgmlkgnY0gmlwhKwSnHCKbXVsdGlhZGRyc5YACASsEpxwBvbQAAoErBKccAb20d0DgnJzhQADAQAAiXNlY3AyNTZrMaECkHNOQvr7eARiRreZxWOjwxlYDFRn7FtfBHv_Aju0JDSDdGNwgvbQg3VkcIL20oV3YWt1Mgk"}'
2026-02-25 15:25:57.919 INFO [src.node.waku_node] REST service is ready !!
2026-02-25 15:25:57.921 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63183/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.161.115/tcp/33594/p2p/16Uiu2HAmQQ3XY4fTraTSTYKFtrD2ddmHKi6z9vt1g1ZKJY38W4w1"]'
2026-02-25 15:25:57.924 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:57.924 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63183/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.214.59/tcp/55177/p2p/16Uiu2HAmGZugaf6yUZY2LRQZQYLEAm5y14hC3aRQvnTt2soRPKrM"]'
2026-02-25 15:25:57.948 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:57.949 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63183/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.64.84/tcp/27695/p2p/16Uiu2HAkwuNPAxWZNCRWxtsJwx7BHNLRNArE2oiXr5bYWptzCW5S"]'
2026-02-25 15:25:57.974 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:57.975 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63183/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.71.162/tcp/51946/p2p/16Uiu2HAmVnVQy9WX6RaY7Pq5hFEwAaSfDtA1gZjKvP84xpDGJtWh"]'
2026-02-25 15:25:57.998 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.001 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33593/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-02-25 15:25:58.008 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.008 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:55176/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-02-25 15:25:58.013 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.013 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27694/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-02-25 15:25:58.017 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.019 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51945/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-02-25 15:25:58.024 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.024 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63183/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-02-25 15:25:58.031 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.032 DEBUG [src.steps.light_push] Lightpushing message
2026-02-25 15:25:58.033 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63183/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/0", "message": {"payload": "TGlnaHQgcHVzaCB3b3JrcyEh", "contentTopic": "/myapp/1/latest/proto", "timestamp": 1772033158032537466}}'
2026-02-25 15:25:58.054 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-25 15:25:58.056 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-02-25 15:25:58.157 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-02-25 15:25:58.157 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33593/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:58.160 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772033158032537466,"ephemeral":false,"proof":""}]'
2026-02-25 15:25:58.162 DEBUG [src.steps.light_push] Checking that peer NODE_2:wakuorg/nwaku:latest can find the lightpushed message
2026-02-25 15:25:58.162 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55176/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:58.164 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772033158032537466,"ephemeral":false,"proof":""}]'
2026-02-25 15:25:58.166 DEBUG [src.steps.light_push] Checking that peer NODE_3:wakuorg/nwaku:latest can find the lightpushed message
2026-02-25 15:25:58.166 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27694/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:58.168 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772033158032537466,"ephemeral":false,"proof":""}]'
2026-02-25 15:25:58.169 DEBUG [src.steps.light_push] Checking that peer NODE_4:wakuorg/nwaku:latest can find the lightpushed message
2026-02-25 15:25:58.170 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51945/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:58.172 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772033158032537466,"ephemeral":false,"proof":""}]'
2026-02-25 15:25:58.173 DEBUG [src.steps.light_push] Checking that peer NODE_5:wakuorg/nwaku:latest can find the lightpushed message
2026-02-25 15:25:58.174 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63183/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-02-25 15:25:58.176 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772033158032537466,"ephemeral":false,"proof":""}]'
2026-02-25 15:25:58.178 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-02-25 15:25:58.179 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-02-25 15:25:58.179 DEBUG [src.node.waku_node] Stopping container with id dddd3052876a
2026-02-25 15:25:58.772 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 15:25:58.772 DEBUG [src.node.waku_node] Stopping container with id 17624d8a4f4c
2026-02-25 15:25:59.403 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 15:25:59.404 DEBUG [src.node.waku_node] Stopping container with id 77b5607c70fd
2026-02-25 15:25:59.586 ERROR [src.node.docker_mananger] Max retries reached for container dddd3052876a. Exiting log stream.
2026-02-25 15:26:00.042 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 15:26:00.044 DEBUG [src.node.waku_node] Stopping container with id c4aa1ece260a
2026-02-25 15:26:00.218 ERROR [src.node.docker_mananger] Max retries reached for container 17624d8a4f4c. Exiting log stream.
2026-02-25 15:26:00.624 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 15:26:00.624 DEBUG [src.node.waku_node] Stopping container with id 2ba8a24aa691
2026-02-25 15:26:00.919 ERROR [src.node.docker_mananger] Max retries reached for container 77b5607c70fd. Exiting log stream.
2026-02-25 15:26:01.177 DEBUG [src.node.waku_node] Container stopped.
2026-02-25 15:26:01.178 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-02-25 15:26:01.193 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 15:26:01.208 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 15:26:01.235 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 15:26:01.250 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-25 15:26:01.265 DEBUG [src.node.docker_mananger] No errors found in the waku logs.