176 lines
32 KiB
Plaintext

2026-03-09 04:46:54.519 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-09 04:46:54.519 DEBUG [tests.conftest] Running test: test_auto_shard_relay with id: 2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746
2026-03-09 04:46:54.520 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-09 04:46:54.520 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-09 04:46:54.520 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-09 04:46:54.527 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:46:54.528 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:54.528 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:46:54.528 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:46:54.529 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:46:54.529 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.59.140
2026-03-09 04:46:54.530 DEBUG [src.node.docker_mananger] Generated ports ['6576', '6577', '6578', '6579', '6580']
2026-03-09 04:46:54.530 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:46:54.530 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:46:54.530 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:46:54.530 DEBUG [src.node.docker_mananger] docker run -i -t -p 6576:6576 -p 6577:6577 -p 6578:6578 -p 6579:6579 -p 6580:6580 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6578 --rest-port=6576 --tcp-port=6577 --discv5-udp-port=6579 --rest-address=0.0.0.0 --nat=extip:172.18.59.140 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=16fbc98814d230b0ebaada15cfa1bad42e8efc7deb083ebceaa0846043c577a0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6580 --metrics-logging=true --relay=true --filter=true --content-topic=/myapp/1/latest/proto
2026-03-09 04:46:54.727 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.59.140 waku e819f4626c8f8cf71d6ae6e4c3ea84678c70c9f96507e6f0f9d5762ac21246e1
2026-03-09 04:46:54.762 DEBUG [src.node.docker_mananger] Container started with ID e819f4626c8f. Setting up logs at ./log/docker/node1_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:54.763 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 6576
2026-03-09 04:46:54.766 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:46:54.769 ERROR [src.node.docker_mananger] Max retries reached for container 441446cb5892. Exiting log stream.
2026-03-09 04:46:55.320 ERROR [src.node.docker_mananger] Max retries reached for container d63719a8fc96. Exiting log stream.
2026-03-09 04:46:55.768 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6576/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:55.773 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:46:55.774 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:46:55.774 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6576/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:55.777 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.59.140/tcp/6577/p2p/16Uiu2HAmEr2nx82qV67spQNQ4yaGCmJYMNaiURPuzUR4JUq3YMxF","/ip4/172.18.59.140/tcp/6578/ws/p2p/16Uiu2HAmEr2nx82qV67spQNQ4yaGCmJYMNaiURPuzUR4JUq3YMxF"],"enrUri":"enr:-L24QC-PVG91b3kMPWQTIMMdcaSBrUOJoMQxSXiRhG2ztdtSJhN6z2eU-baEBRGGzAkcxUF8OXcHTRqKFHlUVWAmWxECgmlkgnY0gmlwhKwSO4yKbXVsdGlhZGRyc5YACASsEjuMBhmxAAoErBI7jAYZst0DgnJzhQACAQAAiXNlY3AyNTZrMaEDIIuGQKgRdJqZrQObua_HwyeXqFpwFbKXaIjXlIpTKIyDdGNwghmxg3VkcIIZs4V3YWt1MgU"}'
2026-03-09 04:46:55.777 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:46:55.785 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:46:55.785 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:55.785 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:46:55.785 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:46:55.786 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:46:55.787 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.211.91
2026-03-09 04:46:55.787 DEBUG [src.node.docker_mananger] Generated ports ['3127', '3128', '3129', '3130', '3131']
2026-03-09 04:46:55.787 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:46:55.787 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:46:55.787 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:46:55.787 DEBUG [src.node.docker_mananger] docker run -i -t -p 3127:3127 -p 3128:3128 -p 3129:3129 -p 3130:3130 -p 3131:3131 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=3129 --rest-port=3127 --tcp-port=3128 --discv5-udp-port=3130 --rest-address=0.0.0.0 --nat=extip:172.18.211.91 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=fe063369bbec0662c56bf7c2c8f7338fe5d8e2217c1db7bfcd8f7ed4aca6d90c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=3131 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC-PVG91b3kMPWQTIMMdcaSBrUOJoMQxSXiRhG2ztdtSJhN6z2eU-baEBRGGzAkcxUF8OXcHTRqKFHlUVWAmWxECgmlkgnY0gmlwhKwSO4yKbXVsdGlhZGRyc5YACASsEjuMBhmxAAoErBI7jAYZst0DgnJzhQACAQAAiXNlY3AyNTZrMaEDIIuGQKgRdJqZrQObua_HwyeXqFpwFbKXaIjXlIpTKIyDdGNwghmxg3VkcIIZs4V3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-09 04:46:55.986 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.211.91 waku b4fe4b6712c0070e6b3cfd2b865003cc7be0dafc0efa361bd73e57bcc4297267
2026-03-09 04:46:56.022 DEBUG [src.node.docker_mananger] Container started with ID b4fe4b6712c0. Setting up logs at ./log/docker/node2_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:56.022 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 3127
2026-03-09 04:46:56.022 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:46:57.023 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3127/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:57.027 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-09 04:46:57.128 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3127/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:57.131 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:46:57.131 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:46:57.131 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3127/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:57.134 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.211.91/tcp/3128/p2p/16Uiu2HAkvgcV3Qa9sPbL8czYas4i6JqxCJu78hzXUiUJu6ZjR2Fg","/ip4/172.18.211.91/tcp/3129/ws/p2p/16Uiu2HAkvgcV3Qa9sPbL8czYas4i6JqxCJu78hzXUiUJu6ZjR2Fg"],"enrUri":"enr:-L24QHskmW6nZIpdeHIndiE_p-27f9ka3s7ymN8_RiHOY7O-UABo97FciwRAiMov2KGvU23c25Paz2K8L_AzNA98ToACgmlkgnY0gmlwhKwS01uKbXVsdGlhZGRyc5YACASsEtNbBgw4AAoErBLTWwYMOd0DgnJzhQACAQAAiXNlY3AyNTZrMaECEq9ySou4ciJeGgRWxwuUtxfeWMd4MNYQT5ItDHZOCReDdGNwggw4g3VkcIIMOoV3YWt1MgE"}'
2026-03-09 04:46:57.134 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:46:57.134 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3127/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.59.140/tcp/6577/p2p/16Uiu2HAmEr2nx82qV67spQNQ4yaGCmJYMNaiURPuzUR4JUq3YMxF"]'
2026-03-09 04:46:57.156 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:46:57.169 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:46:57.169 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:57.171 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:46:57.172 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:46:57.174 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:46:57.175 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.137.123
2026-03-09 04:46:57.175 DEBUG [src.node.docker_mananger] Generated ports ['15129', '15130', '15131', '15132', '15133']
2026-03-09 04:46:57.175 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:46:57.176 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:46:57.176 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:46:57.176 DEBUG [src.node.docker_mananger] docker run -i -t -p 15129:15129 -p 15130:15130 -p 15131:15131 -p 15132:15132 -p 15133:15133 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=15131 --rest-port=15129 --tcp-port=15130 --discv5-udp-port=15132 --rest-address=0.0.0.0 --nat=extip:172.18.137.123 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=6cfd4f915a3bcd5bfad4cd8b5dcf11a4f4d9a910ec42baeccfddc4aadb26abcf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=15133 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC-PVG91b3kMPWQTIMMdcaSBrUOJoMQxSXiRhG2ztdtSJhN6z2eU-baEBRGGzAkcxUF8OXcHTRqKFHlUVWAmWxECgmlkgnY0gmlwhKwSO4yKbXVsdGlhZGRyc5YACASsEjuMBhmxAAoErBI7jAYZst0DgnJzhQACAQAAiXNlY3AyNTZrMaEDIIuGQKgRdJqZrQObua_HwyeXqFpwFbKXaIjXlIpTKIyDdGNwghmxg3VkcIIZs4V3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-09 04:46:57.375 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.137.123 waku 753c179d388e9ff65b9be1938ba7adc46eb5392eaf28409a978924996633c8a4
2026-03-09 04:46:57.415 DEBUG [src.node.docker_mananger] Container started with ID 753c179d388e. Setting up logs at ./log/docker/node3_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:57.415 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 15129
2026-03-09 04:46:57.415 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:46:58.416 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15129/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:58.474 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-09 04:46:58.575 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15129/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:58.578 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:46:58.578 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:46:58.578 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15129/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:58.581 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.137.123/tcp/15130/p2p/16Uiu2HAmER7ohoMNEwNya6Fg77RYp4im2Cf3itB2WbpSGn6zC5dA","/ip4/172.18.137.123/tcp/15131/ws/p2p/16Uiu2HAmER7ohoMNEwNya6Fg77RYp4im2Cf3itB2WbpSGn6zC5dA"],"enrUri":"enr:-L24QMuRCH--Dd7kRxHTUrMQBnsTNCZkoAoVTeo0FJRq9-V2EaPw65cz5fFG48j_4f_ZOsdCqUTobmGvx4e8yLHHVscCgmlkgnY0gmlwhKwSiXuKbXVsdGlhZGRyc5YACASsEol7BjsaAAoErBKJewY7G90DgnJzhQACAQAAiXNlY3AyNTZrMaEDGimsIihA6W3ZmMQt5MXjDX348rJDGI0GKUl6E444RumDdGNwgjsag3VkcII7HIV3YWt1MgE"}'
2026-03-09 04:46:58.581 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:46:58.581 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:15129/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.59.140/tcp/6577/p2p/16Uiu2HAmEr2nx82qV67spQNQ4yaGCmJYMNaiURPuzUR4JUq3YMxF"]'
2026-03-09 04:46:58.623 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:46:58.635 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:46:58.635 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:58.635 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:46:58.635 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:46:58.640 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:46:58.641 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.201.146
2026-03-09 04:46:58.641 DEBUG [src.node.docker_mananger] Generated ports ['15763', '15764', '15765', '15766', '15767']
2026-03-09 04:46:58.641 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:46:58.641 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:46:58.642 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:46:58.642 DEBUG [src.node.docker_mananger] docker run -i -t -p 15763:15763 -p 15764:15764 -p 15765:15765 -p 15766:15766 -p 15767:15767 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=15765 --rest-port=15763 --tcp-port=15764 --discv5-udp-port=15766 --rest-address=0.0.0.0 --nat=extip:172.18.201.146 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=4e715780a05bddba6c7e1adb15512ed251f1fb4dfd8f78e7ee5ff12e9642fd90 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=15767 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC-PVG91b3kMPWQTIMMdcaSBrUOJoMQxSXiRhG2ztdtSJhN6z2eU-baEBRGGzAkcxUF8OXcHTRqKFHlUVWAmWxECgmlkgnY0gmlwhKwSO4yKbXVsdGlhZGRyc5YACASsEjuMBhmxAAoErBI7jAYZst0DgnJzhQACAQAAiXNlY3AyNTZrMaEDIIuGQKgRdJqZrQObua_HwyeXqFpwFbKXaIjXlIpTKIyDdGNwghmxg3VkcIIZs4V3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-09 04:46:58.844 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.201.146 waku 3820d9306c8531f46a1229d2a567f19a59450d88a5825173d68b2bec86ad6307
2026-03-09 04:46:58.879 DEBUG [src.node.docker_mananger] Container started with ID 3820d9306c85. Setting up logs at ./log/docker/node4_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:46:58.880 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 15763
2026-03-09 04:46:58.880 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:46:59.881 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15763/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:59.883 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-09 04:46:59.983 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15763/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:59.986 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:46:59.986 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:46:59.987 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15763/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:46:59.989 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.201.146/tcp/15764/p2p/16Uiu2HAm5F79q25GqwyCeEvuaTci3qzaDWu85Xxz7NmcQfxkx6sE","/ip4/172.18.201.146/tcp/15765/ws/p2p/16Uiu2HAm5F79q25GqwyCeEvuaTci3qzaDWu85Xxz7NmcQfxkx6sE"],"enrUri":"enr:-L24QHdFNJoK3daCHDNUYzD4QEh2haZ2BkRoiIJKJfk8fDt-PERNFLinpXMd6jlEZ_ar_oC7g-N4u1qrVwciOUON-mMCgmlkgnY0gmlwhKwSyZKKbXVsdGlhZGRyc5YACASsEsmSBj2UAAoErBLJkgY9ld0DgnJzhQACAQAAiXNlY3AyNTZrMaECkd_xjwzrPHz6d-8gzqTMTQjBFRscrYYHYrXozS-ncL2DdGNwgj2Ug3VkcII9loV3YWt1MgE"}'
2026-03-09 04:46:59.989 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:46:59.990 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:15763/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.59.140/tcp/6577/p2p/16Uiu2HAmEr2nx82qV67spQNQ4yaGCmJYMNaiURPuzUR4JUq3YMxF"]'
2026-03-09 04:47:00.016 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:00.040 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:47:00.040 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:47:00.040 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:47:00.041 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:47:00.045 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:47:00.045 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.103.101
2026-03-09 04:47:00.047 DEBUG [src.node.docker_mananger] Generated ports ['41042', '41043', '41044', '41045', '41046']
2026-03-09 04:47:00.048 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:47:00.049 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:47:00.049 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:47:00.050 DEBUG [src.node.docker_mananger] docker run -i -t -p 41042:41042 -p 41043:41043 -p 41044:41044 -p 41045:41045 -p 41046:41046 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=41044 --rest-port=41042 --tcp-port=41043 --discv5-udp-port=41045 --rest-address=0.0.0.0 --nat=extip:172.18.103.101 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=64bfbc60cfec9fe1f4d7fcaee6cec46edcee69f16ac4a8a386efcd321bbc4bf0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=41046 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC-PVG91b3kMPWQTIMMdcaSBrUOJoMQxSXiRhG2ztdtSJhN6z2eU-baEBRGGzAkcxUF8OXcHTRqKFHlUVWAmWxECgmlkgnY0gmlwhKwSO4yKbXVsdGlhZGRyc5YACASsEjuMBhmxAAoErBI7jAYZst0DgnJzhQACAQAAiXNlY3AyNTZrMaEDIIuGQKgRdJqZrQObua_HwyeXqFpwFbKXaIjXlIpTKIyDdGNwghmxg3VkcIIZs4V3YWt1MgU --content-topic=/myapp/1/latest/proto
2026-03-09 04:47:00.266 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.103.101 waku 45ec511fb5db38c4647b4940738ef0a593c881a0a402451f90a20731707c32cb
2026-03-09 04:47:00.301 DEBUG [src.node.docker_mananger] Container started with ID 45ec511fb5db. Setting up logs at ./log/docker/node5_2026-03-09_04-46-54__9e57ca31-c6d8-431c-891f-1c77998b3746__wakuorg_nwaku:latest.log
2026-03-09 04:47:00.301 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 41042
2026-03-09 04:47:00.301 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:47:01.302 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41042/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.307 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:47:01.307 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:47:01.307 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41042/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.318 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.103.101/tcp/41043/p2p/16Uiu2HAmVfDLiD9YhLq1w3wB4rdHRPZpbR45XtBomtHT6uJLiCzQ","/ip4/172.18.103.101/tcp/41044/ws/p2p/16Uiu2HAmVfDLiD9YhLq1w3wB4rdHRPZpbR45XtBomtHT6uJLiCzQ"],"enrUri":"enr:-L24QI03ueYzyxck4IQS6Jaa5e9rihwox0ii5wJpSUJZuREIMb65p9yOUjRXZIvnk7Ryz_Q8yelJUmqz-foAWDiDBP4CgmlkgnY0gmlwhKwSZ2WKbXVsdGlhZGRyc5YACASsEmdlBqBTAAoErBJnZQagVN0DgnJzhQACAQAAiXNlY3AyNTZrMaED_KVdmQNRoQ8Sdmvept3RmsS6EbAxn8bedF_Ao_VWlUWDdGNwgqBTg3VkcIKgVYV3YWt1MgE"}'
2026-03-09 04:47:01.319 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:47:01.319 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41042/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.59.140/tcp/6577/p2p/16Uiu2HAmEr2nx82qV67spQNQ4yaGCmJYMNaiURPuzUR4JUq3YMxF"]'
2026-03-09 04:47:01.353 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.355 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6576/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-09 04:47:01.369 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.370 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3127/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-09 04:47:01.379 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.379 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:15129/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-09 04:47:01.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.384 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:15763/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-09 04:47:01.390 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.391 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41042/relay/v1/auto/subscriptions" -H "Content-Type: application/json" -d '["/myapp/1/latest/proto"]'
2026-03-09 04:47:01.401 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.403 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6576/relay/v1/auto/messages" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-09 04:47:01.423 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:47:01.424 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-09 04:47:01.525 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-09 04:47:01.525 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6576/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.528 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773031621403143193,"ephemeral":false,"proof":""}]'
2026-03-09 04:47:01.529 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-09 04:47:01.529 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3127/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.532 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773031621403143193,"ephemeral":false,"proof":""}]'
2026-03-09 04:47:01.533 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-09 04:47:01.534 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15129/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.536 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773031621403143193,"ephemeral":false,"proof":""}]'
2026-03-09 04:47:01.537 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-09 04:47:01.538 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:15763/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.540 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773031621403143193,"ephemeral":false,"proof":""}]'
2026-03-09 04:47:01.541 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-09 04:47:01.542 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41042/relay/v1/auto/messages/%2Fmyapp%2F1%2Flatest%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:47:01.544 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773031621403143193,"ephemeral":false,"proof":""}]'
2026-03-09 04:47:01.547 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-09 04:47:01.548 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-09 04:47:01.548 DEBUG [src.node.waku_node] Stopping container with id e819f4626c8f
2026-03-09 04:47:02.220 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:47:02.222 DEBUG [src.node.waku_node] Stopping container with id b4fe4b6712c0
2026-03-09 04:47:02.810 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:47:02.812 DEBUG [src.node.waku_node] Stopping container with id 753c179d388e
2026-03-09 04:47:03.034 ERROR [src.node.docker_mananger] Max retries reached for container e819f4626c8f. Exiting log stream.
2026-03-09 04:47:03.391 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:47:03.392 DEBUG [src.node.waku_node] Stopping container with id 3820d9306c85
2026-03-09 04:47:03.629 ERROR [src.node.docker_mananger] Max retries reached for container b4fe4b6712c0. Exiting log stream.
2026-03-09 04:47:03.993 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:47:03.995 DEBUG [src.node.waku_node] Stopping container with id 45ec511fb5db
2026-03-09 04:47:04.204 ERROR [src.node.docker_mananger] Max retries reached for container 753c179d388e. Exiting log stream.
2026-03-09 04:47:04.576 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:47:04.580 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-09 04:47:04.631 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:47:04.661 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:47:04.696 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:47:04.730 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:47:04.767 DEBUG [src.node.docker_mananger] No errors found in the waku logs.