2026-03-18 09:27:31 +00:00

170 lines
33 KiB
Plaintext

2026-03-18 09:13:09.751 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-18 09:13:09.752 DEBUG [tests.conftest] Running test: test_auto_shard_relay with id: 2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689
2026-03-18 09:13:09.752 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-18 09:13:09.752 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-18 09:13:09.752 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-18 09:13:09.760 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:13:09.760 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:09.760 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:13:09.760 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:13:09.762 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:13:09.762 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.236.123
2026-03-18 09:13:09.762 DEBUG [src.node.docker_mananger] Generated ports ['19030', '19031', '19032', '19033', '19034']
2026-03-18 09:13:09.762 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:13:09.762 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:13:09.762 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:13:09.762 DEBUG [src.node.docker_mananger] docker run -i -t -p 19030:19030 -p 19031:19031 -p 19032:19032 -p 19033:19033 -p 19034:19034 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=19032 --rest-port=19030 --tcp-port=19031 --discv5-udp-port=19033 --rest-address=0.0.0.0 --nat=extip:172.18.236.123 --peer-exchange=true --discv5-discovery=true --cluster-id=199 --nodekey=812dae5a19ec2111dcff35abcc2fdf28fc3efcaca52fabacaedc5534adf5cf38 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=19034 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto
2026-03-18 09:13:09.959 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.236.123 waku fca9189d98e7b5c2e8a7169ff618cfcab7e31416f9629dd1bbd8a0273d3a34fb
2026-03-18 09:13:09.982 ERROR [src.node.docker_mananger] Max retries reached for container 6ca1372c9191. Exiting log stream.
2026-03-18 09:13:09.995 DEBUG [src.node.docker_mananger] Container started with ID fca9189d98e7. Setting up logs at ./log/docker/node1_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:09.995 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 19030
2026-03-18 09:13:09.995 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:13:10.550 ERROR [src.node.docker_mananger] Max retries reached for container 70d8672d5b17. Exiting log stream.
2026-03-18 09:13:10.996 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19030/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:10.998 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:13:10.999 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:13:10.999 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19030/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:11.001 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.236.123/tcp/19031/p2p/16Uiu2HAmA9nQGE6EbXp48jU31mWYUMCntAEaBgFVQytcHHZcgcDC","/ip4/172.18.236.123/tcp/19032/ws/p2p/16Uiu2HAmA9nQGE6EbXp48jU31mWYUMCntAEaBgFVQytcHHZcgcDC"],"enrUri":"enr:-L24QMuODD4wCgQ8A3-8aZND3BTrhpQb3RqGVvnXVzAt8eUtODMwH3Omd4Dbe8R9QrFy-RVpLVtJO2VIhGV4dfy9o28CgmlkgnY0gmlwhKwS7HuKbXVsdGlhZGRyc5YACASsEux7BkpXAAoErBLsewZKWN0DgnJzhQDHAQAAiXNlY3AyNTZrMaEC2s1Fr2wzSk3lvqAl4Tc2VABsd9-CckyJzZO-llKSXPeDdGNwgkpXg3VkcIJKWYV3YWt1MgU"}'
2026-03-18 09:13:11.001 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:13:11.008 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:13:11.008 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:11.008 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:13:11.008 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:13:11.010 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:13:11.010 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.226.119
2026-03-18 09:13:11.010 DEBUG [src.node.docker_mananger] Generated ports ['27905', '27906', '27907', '27908', '27909']
2026-03-18 09:13:11.010 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:13:11.010 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:13:11.011 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:13:11.011 DEBUG [src.node.docker_mananger] docker run -i -t -p 27905:27905 -p 27906:27906 -p 27907:27907 -p 27908:27908 -p 27909:27909 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=27907 --rest-port=27905 --tcp-port=27906 --discv5-udp-port=27908 --rest-address=0.0.0.0 --nat=extip:172.18.226.119 --peer-exchange=true --discv5-discovery=true --cluster-id=199 --nodekey=99e02d3eebb96cd7fe6aac2f0e5ffa8e13f5ec8d0759cfdd52efb1cf8cef9c6f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=27909 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMuODD4wCgQ8A3-8aZND3BTrhpQb3RqGVvnXVzAt8eUtODMwH3Omd4Dbe8R9QrFy-RVpLVtJO2VIhGV4dfy9o28CgmlkgnY0gmlwhKwS7HuKbXVsdGlhZGRyc5YACASsEux7BkpXAAoErBLsewZKWN0DgnJzhQDHAQAAiXNlY3AyNTZrMaEC2s1Fr2wzSk3lvqAl4Tc2VABsd9-CckyJzZO-llKSXPeDdGNwgkpXg3VkcIJKWYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-18 09:13:11.222 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.226.119 waku b70297a64515237ec93ba1dbb31c754664183c9bb9f34d2b0630d5cbe01bc679
2026-03-18 09:13:11.256 DEBUG [src.node.docker_mananger] Container started with ID b70297a64515. Setting up logs at ./log/docker/node2_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:11.256 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 27905
2026-03-18 09:13:11.257 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:13:12.260 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27905/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:12.286 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:13:12.287 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:13:12.288 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27905/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:12.296 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.226.119/tcp/27906/p2p/16Uiu2HAkuZ9xLow7vonYGEsfrWexqqgLNA9Gx8T2bEJzVPudhEQf","/ip4/172.18.226.119/tcp/27907/ws/p2p/16Uiu2HAkuZ9xLow7vonYGEsfrWexqqgLNA9Gx8T2bEJzVPudhEQf"],"enrUri":"enr:-L24QC-rR8VkwBTCILwGJW4ECnBl9xOBcB25elJfT3GssniPeIMbwnCBo5MwGTTctnT1A2UsV765QvZmcIAKHCwveMACgmlkgnY0gmlwhKwS4neKbXVsdGlhZGRyc5YACASsEuJ3Bm0CAAoErBLidwZtA90DgnJzhQDHAQAAiXNlY3AyNTZrMaECAeqyc3Rk4vXUyfXzBkohV6CQyE3lL19bQNOHaIgnF7CDdGNwgm0Cg3VkcIJtBIV3YWt1MgE"}'
2026-03-18 09:13:12.300 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:13:12.300 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27905/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.236.123/tcp/19031/p2p/16Uiu2HAmA9nQGE6EbXp48jU31mWYUMCntAEaBgFVQytcHHZcgcDC"]'
2026-03-18 09:13:12.305 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:12.312 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:13:12.312 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:12.312 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:13:12.312 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:13:12.313 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:13:12.313 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.38.44
2026-03-18 09:13:12.314 DEBUG [src.node.docker_mananger] Generated ports ['61823', '61824', '61825', '61826', '61827']
2026-03-18 09:13:12.314 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:13:12.314 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:13:12.314 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:13:12.314 DEBUG [src.node.docker_mananger] docker run -i -t -p 61823:61823 -p 61824:61824 -p 61825:61825 -p 61826:61826 -p 61827:61827 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=61825 --rest-port=61823 --tcp-port=61824 --discv5-udp-port=61826 --rest-address=0.0.0.0 --nat=extip:172.18.38.44 --peer-exchange=true --discv5-discovery=true --cluster-id=199 --nodekey=24ef76fcea9bd3b3d23946fc734b0c48edcbe7a8e5aface4fecfe6e7b0cbcf84 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=61827 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMuODD4wCgQ8A3-8aZND3BTrhpQb3RqGVvnXVzAt8eUtODMwH3Omd4Dbe8R9QrFy-RVpLVtJO2VIhGV4dfy9o28CgmlkgnY0gmlwhKwS7HuKbXVsdGlhZGRyc5YACASsEux7BkpXAAoErBLsewZKWN0DgnJzhQDHAQAAiXNlY3AyNTZrMaEC2s1Fr2wzSk3lvqAl4Tc2VABsd9-CckyJzZO-llKSXPeDdGNwgkpXg3VkcIJKWYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-18 09:13:12.510 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.38.44 waku d1a9647b50dfd1172d516ff6d714d7e062d3edf010d272d7de139089d48f46e5
2026-03-18 09:13:12.545 DEBUG [src.node.docker_mananger] Container started with ID d1a9647b50df. Setting up logs at ./log/docker/node3_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:12.545 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 61823
2026-03-18 09:13:12.545 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:13:13.546 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61823/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:13.567 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:13:13.568 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:13:13.570 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61823/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:13.576 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.38.44/tcp/61824/p2p/16Uiu2HAkzt7ahQvitFM3e4c66P98DEVZgGtKVsCq42voV2TwGDfq","/ip4/172.18.38.44/tcp/61825/ws/p2p/16Uiu2HAkzt7ahQvitFM3e4c66P98DEVZgGtKVsCq42voV2TwGDfq"],"enrUri":"enr:-L24QFrA3E88O1WC_fKE8LpKKXBIFbwkCNjZ4CvMXcOgR7Sxc87oPE3Zbr4bndKoGQHAN9ZHsOUqjv5tUfnibSAKPnICgmlkgnY0gmlwhKwSJiyKbXVsdGlhZGRyc5YACASsEiYsBvGAAAoErBImLAbxgd0DgnJzhQDHAQAAiXNlY3AyNTZrMaECURB71FZ-ZEKRYt_h65g-bgoVY7uUqNRvPaviuOCYurSDdGNwgvGAg3VkcILxgoV3YWt1MgE"}'
2026-03-18 09:13:13.577 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:13:13.577 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:61823/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.236.123/tcp/19031/p2p/16Uiu2HAmA9nQGE6EbXp48jU31mWYUMCntAEaBgFVQytcHHZcgcDC"]'
2026-03-18 09:13:13.580 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:13.587 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:13:13.588 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:13.588 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:13:13.588 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:13:13.589 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:13:13.589 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.50.230
2026-03-18 09:13:13.590 DEBUG [src.node.docker_mananger] Generated ports ['17290', '17291', '17292', '17293', '17294']
2026-03-18 09:13:13.590 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:13:13.590 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:13:13.590 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:13:13.590 DEBUG [src.node.docker_mananger] docker run -i -t -p 17290:17290 -p 17291:17291 -p 17292:17292 -p 17293:17293 -p 17294:17294 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17292 --rest-port=17290 --tcp-port=17291 --discv5-udp-port=17293 --rest-address=0.0.0.0 --nat=extip:172.18.50.230 --peer-exchange=true --discv5-discovery=true --cluster-id=199 --nodekey=6ccd2c5eea6eecd4bc0cfc5da9c762712f36d47eafad3538bf979bdadbd0a2cd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17294 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMuODD4wCgQ8A3-8aZND3BTrhpQb3RqGVvnXVzAt8eUtODMwH3Omd4Dbe8R9QrFy-RVpLVtJO2VIhGV4dfy9o28CgmlkgnY0gmlwhKwS7HuKbXVsdGlhZGRyc5YACASsEux7BkpXAAoErBLsewZKWN0DgnJzhQDHAQAAiXNlY3AyNTZrMaEC2s1Fr2wzSk3lvqAl4Tc2VABsd9-CckyJzZO-llKSXPeDdGNwgkpXg3VkcIJKWYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-18 09:13:13.799 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.50.230 waku 20c49c37a8d4b9dc5edc6bda598950181f049647d1e089f8aa87958481728e46
2026-03-18 09:13:13.834 DEBUG [src.node.docker_mananger] Container started with ID 20c49c37a8d4. Setting up logs at ./log/docker/node4_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:13.835 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 17290
2026-03-18 09:13:13.835 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:13:14.836 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17290/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:14.850 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:13:14.851 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:13:14.851 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17290/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:14.860 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.50.230/tcp/17291/p2p/16Uiu2HAm7hJGgUPGhxGQ9ZuTtxoBWQwHS3bkeL7o21yWSct9rzuz","/ip4/172.18.50.230/tcp/17292/ws/p2p/16Uiu2HAm7hJGgUPGhxGQ9ZuTtxoBWQwHS3bkeL7o21yWSct9rzuz"],"enrUri":"enr:-L24QP7DKdnPsYYd9hpohBNDS8zLnrUWUaLOWEqFM1cwNu6DM7zypoltQO-RbMCWuXUAirQzEO7Hu7EnErNSqDXN7P4CgmlkgnY0gmlwhKwSMuaKbXVsdGlhZGRyc5YACASsEjLmBkOLAAoErBIy5gZDjN0DgnJzhQDHAQAAiXNlY3AyNTZrMaECtkz9ZB7SsLP_50xkP_J4F-0H-4vSMRz8iKL85okAyE2DdGNwgkOLg3VkcIJDjYV3YWt1MgE"}'
2026-03-18 09:13:14.861 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:13:14.862 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17290/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.236.123/tcp/19031/p2p/16Uiu2HAmA9nQGE6EbXp48jU31mWYUMCntAEaBgFVQytcHHZcgcDC"]'
2026-03-18 09:13:14.864 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:14.871 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0
2026-03-18 09:13:14.871 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:14.872 DEBUG [src.node.waku_node] Starting Node...
2026-03-18 09:13:14.872 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-18 09:13:14.873 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-18 09:13:14.873 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.15.254
2026-03-18 09:13:14.873 DEBUG [src.node.docker_mananger] Generated ports ['41701', '41702', '41703', '41704', '41705']
2026-03-18 09:13:14.873 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-18 09:13:14.873 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-18 09:13:14.874 DEBUG [src.node.waku_node] Using volumes []
2026-03-18 09:13:14.874 DEBUG [src.node.docker_mananger] docker run -i -t -p 41701:41701 -p 41702:41702 -p 41703:41703 -p 41704:41704 -p 41705:41705 harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=41703 --rest-port=41701 --tcp-port=41702 --discv5-udp-port=41704 --rest-address=0.0.0.0 --nat=extip:172.18.15.254 --peer-exchange=true --discv5-discovery=true --cluster-id=199 --nodekey=ed84f6fb0bcb1fda2de876eb48e1070342c11d3fb8639f8b79b7aedda92e9d9a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=41705 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMuODD4wCgQ8A3-8aZND3BTrhpQb3RqGVvnXVzAt8eUtODMwH3Omd4Dbe8R9QrFy-RVpLVtJO2VIhGV4dfy9o28CgmlkgnY0gmlwhKwS7HuKbXVsdGlhZGRyc5YACASsEux7BkpXAAoErBLsewZKWN0DgnJzhQDHAQAAiXNlY3AyNTZrMaEC2s1Fr2wzSk3lvqAl4Tc2VABsd9-CckyJzZO-llKSXPeDdGNwgkpXg3VkcIJKWYV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-18 09:13:15.076 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.15.254 waku b994b545d565c73436580cb12fce01f8f27800cdec82402dfb5c0b00bfe67a8a
2026-03-18 09:13:15.112 DEBUG [src.node.docker_mananger] Container started with ID b994b545d565. Setting up logs at ./log/docker/node5_2026-03-18_09-13-09__37727336-2cba-4bd4-bddb-9ff2acb4d689__harbor.status.im_wakuorg_nwaku:v0.38.0-rc.0.log
2026-03-18 09:13:15.112 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0. REST: 41701
2026-03-18 09:13:15.112 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-18 09:13:16.112 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41701/health" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.128 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-18 09:13:16.128 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-18 09:13:16.128 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41701/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.136 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.15.254/tcp/41702/p2p/16Uiu2HAm9szx2Nsjwe5jDH73RxkMt2M7YUYJvh7sb1kR8JNuM7Bz","/ip4/172.18.15.254/tcp/41703/ws/p2p/16Uiu2HAm9szx2Nsjwe5jDH73RxkMt2M7YUYJvh7sb1kR8JNuM7Bz"],"enrUri":"enr:-L24QE001VgeaBlcpkJj6erzZYBrwKjo0Ljc5McVD4Rz9dM4EjgTDrfdjcIxQjpYtvP2EcE8bgSsNulerEv1brPb6_gCgmlkgnY0gmlwhKwSD_6KbXVsdGlhZGRyc5YACASsEg_-BqLmAAoErBIP_gai590DgnJzhQDHAQAAiXNlY3AyNTZrMaEC1sIquSf_2Qpc6X9X094izl0me2Gur_V87HvwI-_W45WDdGNwgqLmg3VkcIKi6IV3YWt1MgE"}'
2026-03-18 09:13:16.137 INFO [src.node.waku_node] REST service is ready !!
2026-03-18 09:13:16.137 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41701/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.236.123/tcp/19031/p2p/16Uiu2HAmA9nQGE6EbXp48jU31mWYUMCntAEaBgFVQytcHHZcgcDC"]'
2026-03-18 09:13:16.140 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.141 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19030/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-18 09:13:16.143 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.143 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27905/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-18 09:13:16.146 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.146 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:61823/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-18 09:13:16.149 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.149 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17290/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-18 09:13:16.152 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.152 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41701/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-18 09:13:16.154 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.155 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19030/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-18 09:13:16.163 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-18 09:13:16.163 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-18 09:13:16.264 DEBUG [src.steps.sharding] Checking that peer NODE_1:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:13:16.264 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19030/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.266 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773825196155171681,"ephemeral":false,"proof":""}]'
2026-03-18 09:13:16.268 DEBUG [src.steps.sharding] Checking that peer NODE_2:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:13:16.268 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27905/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.270 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773825196155171681,"ephemeral":false,"proof":""}]'
2026-03-18 09:13:16.272 DEBUG [src.steps.sharding] Checking that peer NODE_3:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:13:16.272 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61823/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.274 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773825196155171681,"ephemeral":false,"proof":""}]'
2026-03-18 09:13:16.276 DEBUG [src.steps.sharding] Checking that peer NODE_4:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:13:16.276 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17290/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.278 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773825196155171681,"ephemeral":false,"proof":""}]'
2026-03-18 09:13:16.279 DEBUG [src.steps.sharding] Checking that peer NODE_5:harbor.status.im/wakuorg/nwaku:v0.38.0-rc.0 can find the published message
2026-03-18 09:13:16.279 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41701/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-18 09:13:16.282 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773825196155171681,"ephemeral":false,"proof":""}]'
2026-03-18 09:13:16.284 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-18 09:13:16.285 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-18 09:13:16.286 DEBUG [src.node.waku_node] Stopping container with id fca9189d98e7
2026-03-18 09:13:16.906 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:13:16.907 DEBUG [src.node.waku_node] Stopping container with id b70297a64515
2026-03-18 09:13:17.470 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:13:17.471 DEBUG [src.node.waku_node] Stopping container with id d1a9647b50df
2026-03-18 09:13:17.720 ERROR [src.node.docker_mananger] Max retries reached for container fca9189d98e7. Exiting log stream.
2026-03-18 09:13:18.036 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:13:18.037 DEBUG [src.node.waku_node] Stopping container with id 20c49c37a8d4
2026-03-18 09:13:18.310 ERROR [src.node.docker_mananger] Max retries reached for container b70297a64515. Exiting log stream.
2026-03-18 09:13:18.610 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:13:18.613 DEBUG [src.node.waku_node] Stopping container with id b994b545d565
2026-03-18 09:13:18.850 ERROR [src.node.docker_mananger] Max retries reached for container d1a9647b50df. Exiting log stream.
2026-03-18 09:13:19.205 DEBUG [src.node.waku_node] Container stopped.
2026-03-18 09:13:19.206 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-18 09:13:19.220 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:13:19.233 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:13:19.238 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:13:19.242 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-18 09:13:19.247 DEBUG [src.node.docker_mananger] No errors found in the waku logs.