169 lines
32 KiB
Plaintext

2026-02-14 04:40:03.689 DEBUG [tests.conftest] Running fixture setup: test_id
2026-02-14 04:40:03.690 DEBUG [tests.conftest] Running test: test_auto_shard_relay with id: 2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e
2026-02-14 04:40:03.691 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-02-14 04:40:03.691 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-02-14 04:40:03.691 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-02-14 04:40:03.702 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-14 04:40:03.703 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:03.703 DEBUG [src.node.waku_node] Starting Node...
2026-02-14 04:40:03.704 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-14 04:40:03.705 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-14 04:40:03.705 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.252.109
2026-02-14 04:40:03.706 DEBUG [src.node.docker_mananger] Generated ports ['13489', '13490', '13491', '13492', '13493']
2026-02-14 04:40:03.706 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-14 04:40:03.706 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-14 04:40:03.706 DEBUG [src.node.waku_node] Using volumes []
2026-02-14 04:40:03.706 DEBUG [src.node.docker_mananger] docker run -i -t -p 13489:13489 -p 13490:13490 -p 13491:13491 -p 13492:13492 -p 13493:13493 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=13491 --rest-port=13489 --tcp-port=13490 --discv5-udp-port=13492 --rest-address=0.0.0.0 --nat=extip:172.18.252.109 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=a9a4d46887eb1cdc02bddaad7ffe63bbafdbaa282ab92a854fd7ebac333a1c5c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=13493 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto
2026-02-14 04:40:03.924 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.252.109 waku fcb170b128a1a9b8b94cdc1a6948ff7982f74115e79bf4a976f571706a8b644d
2026-02-14 04:40:03.977 DEBUG [src.node.docker_mananger] Container started with ID fcb170b128a1. Setting up logs at ./log/docker/node1_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:03.978 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 13489
2026-02-14 04:40:03.978 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-14 04:40:04.246 ERROR [src.node.docker_mananger] Max retries reached for container cfa4466a1c90. Exiting log stream.
2026-02-14 04:40:04.515 ERROR [src.node.docker_mananger] Max retries reached for container fe18fc4a6eff. Exiting log stream.
2026-02-14 04:40:04.979 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13489/health" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:04.982 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-14 04:40:04.982 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-14 04:40:04.983 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13489/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:04.985 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.252.109/tcp/13490/p2p/16Uiu2HAm58BPnmJGJArfRpstN8bk4zAyC9jAuJVFTPvqZrvbbN6g","/ip4/172.18.252.109/tcp/13491/ws/p2p/16Uiu2HAm58BPnmJGJArfRpstN8bk4zAyC9jAuJVFTPvqZrvbbN6g"],"enrUri":"enr:-L24QD-CPfFaalbFkVHCU5Ynz7_--tc3-GnHR8vD3bCNmzCEEStqhF2Gr7QgQzNs_iuj-RxOZO1ElxhlGu1uBkopWuoCgmlkgnY0gmlwhKwS_G2KbXVsdGlhZGRyc5YACASsEvxtBjSyAAoErBL8bQY0s90DgnJzhQACAQAAiXNlY3AyNTZrMaECkBmsQ9z5f5gq2Rf9n0c3eZKGyo_3OQQrdaxb7AY9QQ2DdGNwgjSyg3VkcII0tIV3YWt1MgU"}'
2026-02-14 04:40:04.986 INFO [src.node.waku_node] REST service is ready !!
2026-02-14 04:40:04.993 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-14 04:40:04.993 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:04.993 DEBUG [src.node.waku_node] Starting Node...
2026-02-14 04:40:04.993 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-14 04:40:04.995 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-14 04:40:04.995 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.146.34
2026-02-14 04:40:04.995 DEBUG [src.node.docker_mananger] Generated ports ['64688', '64689', '64690', '64691', '64692']
2026-02-14 04:40:04.995 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-14 04:40:04.995 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-14 04:40:04.995 DEBUG [src.node.waku_node] Using volumes []
2026-02-14 04:40:04.996 DEBUG [src.node.docker_mananger] docker run -i -t -p 64688:64688 -p 64689:64689 -p 64690:64690 -p 64691:64691 -p 64692:64692 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=64690 --rest-port=64688 --tcp-port=64689 --discv5-udp-port=64691 --rest-address=0.0.0.0 --nat=extip:172.18.146.34 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=ba925dfccbdbaddff8bac62fd6e8a9f919505deeada4dd2bb68f3ea1f710e49e --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=64692 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QD-CPfFaalbFkVHCU5Ynz7_--tc3-GnHR8vD3bCNmzCEEStqhF2Gr7QgQzNs_iuj-RxOZO1ElxhlGu1uBkopWuoCgmlkgnY0gmlwhKwS_G2KbXVsdGlhZGRyc5YACASsEvxtBjSyAAoErBL8bQY0s90DgnJzhQACAQAAiXNlY3AyNTZrMaECkBmsQ9z5f5gq2Rf9n0c3eZKGyo_3OQQrdaxb7AY9QQ2DdGNwgjSyg3VkcII0tIV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-02-14 04:40:05.200 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.146.34 waku ef73013b4a54ddc8373859636169e78d5ec64ec9de249d32d53eb2ea229516e1
2026-02-14 04:40:05.251 DEBUG [src.node.docker_mananger] Container started with ID ef73013b4a54. Setting up logs at ./log/docker/node2_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:05.252 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 64688
2026-02-14 04:40:05.252 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-14 04:40:06.253 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64688/health" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:06.261 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-14 04:40:06.264 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-14 04:40:06.264 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64688/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:06.270 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.146.34/tcp/64689/p2p/16Uiu2HAkztaey8GRegEub2CAiUhNyyP7rm2So4sBHFJ2X9UbdQH8","/ip4/172.18.146.34/tcp/64690/ws/p2p/16Uiu2HAkztaey8GRegEub2CAiUhNyyP7rm2So4sBHFJ2X9UbdQH8"],"enrUri":"enr:-L24QHV7Yp_YFmtq4R5cnkDfcuofS8fHencrFrNgb_qvT_8nbXaD6u-FzTvrjz9NIDA_zSxQ-i0F8p0fCoaxjFnfW9ICgmlkgnY0gmlwhKwSkiKKbXVsdGlhZGRyc5YACASsEpIiBvyxAAoErBKSIgb8st0DgnJzhQACAQAAiXNlY3AyNTZrMaECUS8YkUs2KWDY-uGM9VVQpuDL--9blSCw7JrFb-jgA4ODdGNwgvyxg3VkcIL8s4V3YWt1MgE"}'
2026-02-14 04:40:06.271 INFO [src.node.waku_node] REST service is ready !!
2026-02-14 04:40:06.271 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:64688/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.252.109/tcp/13490/p2p/16Uiu2HAm58BPnmJGJArfRpstN8bk4zAyC9jAuJVFTPvqZrvbbN6g"]'
2026-02-14 04:40:06.274 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:06.281 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-14 04:40:06.282 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:06.282 DEBUG [src.node.waku_node] Starting Node...
2026-02-14 04:40:06.282 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-14 04:40:06.283 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-14 04:40:06.283 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.123.174
2026-02-14 04:40:06.284 DEBUG [src.node.docker_mananger] Generated ports ['6933', '6934', '6935', '6936', '6937']
2026-02-14 04:40:06.284 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-14 04:40:06.284 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-14 04:40:06.284 DEBUG [src.node.waku_node] Using volumes []
2026-02-14 04:40:06.284 DEBUG [src.node.docker_mananger] docker run -i -t -p 6933:6933 -p 6934:6934 -p 6935:6935 -p 6936:6936 -p 6937:6937 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6935 --rest-port=6933 --tcp-port=6934 --discv5-udp-port=6936 --rest-address=0.0.0.0 --nat=extip:172.18.123.174 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=c1b9823cecaad8e8a3fd8f94aa1e26b0f94f5f86abef3ec6b22fc857272e11fb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6937 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QD-CPfFaalbFkVHCU5Ynz7_--tc3-GnHR8vD3bCNmzCEEStqhF2Gr7QgQzNs_iuj-RxOZO1ElxhlGu1uBkopWuoCgmlkgnY0gmlwhKwS_G2KbXVsdGlhZGRyc5YACASsEvxtBjSyAAoErBL8bQY0s90DgnJzhQACAQAAiXNlY3AyNTZrMaECkBmsQ9z5f5gq2Rf9n0c3eZKGyo_3OQQrdaxb7AY9QQ2DdGNwgjSyg3VkcII0tIV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-02-14 04:40:06.496 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.123.174 waku 0facf5bbff450751e4ed8eed54eaaac68605873f5ffbf2f938a901d9780468c9
2026-02-14 04:40:06.544 DEBUG [src.node.docker_mananger] Container started with ID 0facf5bbff45. Setting up logs at ./log/docker/node3_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:06.544 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 6933
2026-02-14 04:40:06.545 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-14 04:40:07.545 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6933/health" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:07.554 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-14 04:40:07.555 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-14 04:40:07.555 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6933/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:07.559 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.123.174/tcp/6934/p2p/16Uiu2HAmNBSDtv8juDvL8JuDttMq42EiT4P9qfjDMDYduo9t49TR","/ip4/172.18.123.174/tcp/6935/ws/p2p/16Uiu2HAmNBSDtv8juDvL8JuDttMq42EiT4P9qfjDMDYduo9t49TR"],"enrUri":"enr:-L24QGIg1FfE-3ErMLiJTZQSDw1gCxg9EoiHbrY-VYebDQ7dID5B_STn1J94umB9mRc5VjaEA-K3LywW0PORpgfPPdECgmlkgnY0gmlwhKwSe66KbXVsdGlhZGRyc5YACASsEnuuBhsWAAoErBJ7rgYbF90DgnJzhQACAQAAiXNlY3AyNTZrMaEDjYXcWrCXpSBmMIQ8G0Ehh87F55BmKH5uUVc1cPVFwESDdGNwghsWg3VkcIIbGIV3YWt1MgE"}'
2026-02-14 04:40:07.559 INFO [src.node.waku_node] REST service is ready !!
2026-02-14 04:40:07.560 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6933/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.252.109/tcp/13490/p2p/16Uiu2HAm58BPnmJGJArfRpstN8bk4zAyC9jAuJVFTPvqZrvbbN6g"]'
2026-02-14 04:40:07.563 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:07.570 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-14 04:40:07.570 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:07.570 DEBUG [src.node.waku_node] Starting Node...
2026-02-14 04:40:07.570 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-14 04:40:07.571 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-14 04:40:07.572 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.128.237
2026-02-14 04:40:07.572 DEBUG [src.node.docker_mananger] Generated ports ['60417', '60418', '60419', '60420', '60421']
2026-02-14 04:40:07.572 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-14 04:40:07.572 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-14 04:40:07.572 DEBUG [src.node.waku_node] Using volumes []
2026-02-14 04:40:07.572 DEBUG [src.node.docker_mananger] docker run -i -t -p 60417:60417 -p 60418:60418 -p 60419:60419 -p 60420:60420 -p 60421:60421 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=60419 --rest-port=60417 --tcp-port=60418 --discv5-udp-port=60420 --rest-address=0.0.0.0 --nat=extip:172.18.128.237 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=c62c3beb7fea46cd10bfea1a9dced52b9eecae93ee2d3dcde0bc33c5c4d19177 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=60421 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QD-CPfFaalbFkVHCU5Ynz7_--tc3-GnHR8vD3bCNmzCEEStqhF2Gr7QgQzNs_iuj-RxOZO1ElxhlGu1uBkopWuoCgmlkgnY0gmlwhKwS_G2KbXVsdGlhZGRyc5YACASsEvxtBjSyAAoErBL8bQY0s90DgnJzhQACAQAAiXNlY3AyNTZrMaECkBmsQ9z5f5gq2Rf9n0c3eZKGyo_3OQQrdaxb7AY9QQ2DdGNwgjSyg3VkcII0tIV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-02-14 04:40:07.785 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.128.237 waku bc3442fc2dae0d7fc7f34f2f7ea91265f4751adbfc98e85848a5317000e29d31
2026-02-14 04:40:07.835 DEBUG [src.node.docker_mananger] Container started with ID bc3442fc2dae. Setting up logs at ./log/docker/node4_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:07.836 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 60417
2026-02-14 04:40:07.837 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-14 04:40:08.838 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60417/health" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:08.848 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-14 04:40:08.848 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-14 04:40:08.850 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60417/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:08.855 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.128.237/tcp/60418/p2p/16Uiu2HAmLDhzhzMnBMsiBgspxvPkgdXqmoVjvuTKJkQA8fdd8cT2","/ip4/172.18.128.237/tcp/60419/ws/p2p/16Uiu2HAmLDhzhzMnBMsiBgspxvPkgdXqmoVjvuTKJkQA8fdd8cT2"],"enrUri":"enr:-L24QPjeWHvTe7d5V5bSU-lsLa1rht4U-a7Z9aOST-iFicXqROvdYoeLSLxodFR44wFFCZLXmWuxIRQ6k_uvmNRjeiACgmlkgnY0gmlwhKwSgO2KbXVsdGlhZGRyc5YACASsEoDtBuwCAAoErBKA7QbsA90DgnJzhQACAQAAiXNlY3AyNTZrMaEDcGN7kJ_t0FGdCYvzi_1qeyL7pvRp6hpYu9M9Ofrak6mDdGNwguwCg3VkcILsBIV3YWt1MgE"}'
2026-02-14 04:40:08.855 INFO [src.node.waku_node] REST service is ready !!
2026-02-14 04:40:08.856 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60417/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.252.109/tcp/13490/p2p/16Uiu2HAm58BPnmJGJArfRpstN8bk4zAyC9jAuJVFTPvqZrvbbN6g"]'
2026-02-14 04:40:08.859 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:08.866 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-02-14 04:40:08.866 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:08.867 DEBUG [src.node.waku_node] Starting Node...
2026-02-14 04:40:08.867 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-02-14 04:40:08.868 DEBUG [src.node.docker_mananger] Network waku already exists
2026-02-14 04:40:08.868 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.145.156
2026-02-14 04:40:08.868 DEBUG [src.node.docker_mananger] Generated ports ['31632', '31633', '31634', '31635', '31636']
2026-02-14 04:40:08.869 DEBUG [src.node.waku_node] RLN credentials were not set
2026-02-14 04:40:08.869 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-02-14 04:40:08.869 DEBUG [src.node.waku_node] Using volumes []
2026-02-14 04:40:08.869 DEBUG [src.node.docker_mananger] docker run -i -t -p 31632:31632 -p 31633:31633 -p 31634:31634 -p 31635:31635 -p 31636:31636 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=31634 --rest-port=31632 --tcp-port=31633 --discv5-udp-port=31635 --rest-address=0.0.0.0 --nat=extip:172.18.145.156 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f830a881cb8c1e50e5018bcd1ef546d3a8f0af8c5dffe7eb117c4efcbb1d182a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=31636 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QD-CPfFaalbFkVHCU5Ynz7_--tc3-GnHR8vD3bCNmzCEEStqhF2Gr7QgQzNs_iuj-RxOZO1ElxhlGu1uBkopWuoCgmlkgnY0gmlwhKwS_G2KbXVsdGlhZGRyc5YACASsEvxtBjSyAAoErBL8bQY0s90DgnJzhQACAQAAiXNlY3AyNTZrMaECkBmsQ9z5f5gq2Rf9n0c3eZKGyo_3OQQrdaxb7AY9QQ2DdGNwgjSyg3VkcII0tIV3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-02-14 04:40:09.078 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.145.156 waku 751cb329838bb7f939d501b69c2b468677f15158edce3c50ebb128b6bbb9da73
2026-02-14 04:40:09.126 DEBUG [src.node.docker_mananger] Container started with ID 751cb329838b. Setting up logs at ./log/docker/node5_2026-02-14_04-40-03__3dce3623-fc8b-4da1-8320-f86da901089e__wakuorg_nwaku:latest.log
2026-02-14 04:40:09.126 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 31632
2026-02-14 04:40:09.127 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-02-14 04:40:10.127 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31632/health" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.136 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-02-14 04:40:10.136 INFO [src.node.waku_node] Node protocols are initialized !!
2026-02-14 04:40:10.137 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31632/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.139 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.145.156/tcp/31633/p2p/16Uiu2HAmBFx6XJKvec4coFVDHG1Ht71FwqtKdWFaFqTxtC9YfMpy","/ip4/172.18.145.156/tcp/31634/ws/p2p/16Uiu2HAmBFx6XJKvec4coFVDHG1Ht71FwqtKdWFaFqTxtC9YfMpy"],"enrUri":"enr:-L24QCPIe5DUgZGvL_UPePo7RZC0c56oVWG3fMK_3EjkeTsAdzXEnAfA2YB5WTA3nr7aLb5PGxOytwLuVdezx7xkkbICgmlkgnY0gmlwhKwSkZyKbXVsdGlhZGRyc5YACASsEpGcBnuRAAoErBKRnAZ7kt0DgnJzhQACAQAAiXNlY3AyNTZrMaEC6z1nuhVErn_cg1EdR-J8j2CavGTck4qJ2ZItBTiVbM6DdGNwgnuRg3VkcIJ7k4V3YWt1MgE"}'
2026-02-14 04:40:10.139 INFO [src.node.waku_node] REST service is ready !!
2026-02-14 04:40:10.140 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:31632/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.252.109/tcp/13490/p2p/16Uiu2HAm58BPnmJGJArfRpstN8bk4zAyC9jAuJVFTPvqZrvbbN6g"]'
2026-02-14 04:40:10.142 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.143 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13489/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-02-14 04:40:10.145 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.146 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:64688/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-02-14 04:40:10.148 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.149 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6933/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-02-14 04:40:10.151 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.152 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60417/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-02-14 04:40:10.154 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.155 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:31632/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-02-14 04:40:10.157 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.158 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13489/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-02-14 04:40:10.166 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-02-14 04:40:10.168 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-02-14 04:40:10.271 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-02-14 04:40:10.271 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13489/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.274 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1771044010158135592,"ephemeral":false,"proof":""}]'
2026-02-14 04:40:10.276 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-02-14 04:40:10.276 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64688/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.279 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1771044010158135592,"ephemeral":false,"proof":""}]'
2026-02-14 04:40:10.280 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-02-14 04:40:10.280 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6933/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.283 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1771044010158135592,"ephemeral":false,"proof":""}]'
2026-02-14 04:40:10.284 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-02-14 04:40:10.284 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60417/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.287 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1771044010158135592,"ephemeral":false,"proof":""}]'
2026-02-14 04:40:10.288 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-02-14 04:40:10.288 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:31632/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-02-14 04:40:10.291 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1771044010158135592,"ephemeral":false,"proof":""}]'
2026-02-14 04:40:10.294 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-02-14 04:40:10.296 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-02-14 04:40:10.296 DEBUG [src.node.waku_node] Stopping container with id fcb170b128a1
2026-02-14 04:40:10.580 DEBUG [src.node.waku_node] Container stopped.
2026-02-14 04:40:10.580 DEBUG [src.node.waku_node] Stopping container with id ef73013b4a54
2026-02-14 04:40:10.855 DEBUG [src.node.waku_node] Container stopped.
2026-02-14 04:40:10.857 DEBUG [src.node.waku_node] Stopping container with id 0facf5bbff45
2026-02-14 04:40:11.148 DEBUG [src.node.waku_node] Container stopped.
2026-02-14 04:40:11.148 DEBUG [src.node.waku_node] Stopping container with id bc3442fc2dae
2026-02-14 04:40:11.405 DEBUG [src.node.waku_node] Container stopped.
2026-02-14 04:40:11.405 DEBUG [src.node.waku_node] Stopping container with id 751cb329838b
2026-02-14 04:40:11.450 ERROR [src.node.docker_mananger] Max retries reached for container fcb170b128a1. Exiting log stream.
2026-02-14 04:40:11.662 DEBUG [src.node.waku_node] Container stopped.
2026-02-14 04:40:11.663 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-02-14 04:40:11.692 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-14 04:40:11.703 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-14 04:40:11.714 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-14 04:40:11.717 ERROR [src.node.docker_mananger] Max retries reached for container ef73013b4a54. Exiting log stream.
2026-02-14 04:40:11.726 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-02-14 04:40:11.734 DEBUG [src.node.docker_mananger] No errors found in the waku logs.