198 lines
34 KiB
Plaintext

2026-03-16 05:00:10.588 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-16 05:00:10.588 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6
2026-03-16 05:00:10.589 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-16 05:00:10.589 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-16 05:00:10.589 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-16 05:00:10.597 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-16 05:00:10.597 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:10.597 DEBUG [src.node.waku_node] Starting Node...
2026-03-16 05:00:10.597 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-16 05:00:10.599 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-16 05:00:10.599 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.186.175
2026-03-16 05:00:10.599 DEBUG [src.node.docker_mananger] Generated ports ['5759', '5760', '5761', '5762', '5763']
2026-03-16 05:00:10.599 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-16 05:00:10.600 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-16 05:00:10.600 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-16 05:00:10.600 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-16 05:00:10.600 DEBUG [src.node.waku_node] Using volumes []
2026-03-16 05:00:10.600 DEBUG [src.node.docker_mananger] docker run -i -t -p 5759:5759 -p 5760:5760 -p 5761:5761 -p 5762:5762 -p 5763:5763 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=5761 --rest-port=5759 --tcp-port=5760 --discv5-udp-port=5762 --rest-address=0.0.0.0 --nat=extip:172.18.186.175 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=8b8b0b428dcc9b21abab5e945c7e52ddfceeb5ee0dfd2f6cbac2a79ceff6fbeb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=5763 --metrics-logging=true --relay=true --filter=true
2026-03-16 05:00:10.801 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.186.175 waku f13504672fe32c8669a9e992ae25a89cfa9068d61cf8b2070305831539415909
2026-03-16 05:00:10.825 ERROR [src.node.docker_mananger] Max retries reached for container 214039aefcee. Exiting log stream.
2026-03-16 05:00:10.837 DEBUG [src.node.docker_mananger] Container started with ID f13504672fe3. Setting up logs at ./log/docker/node1_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:10.838 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 5759
2026-03-16 05:00:10.839 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-16 05:00:11.391 ERROR [src.node.docker_mananger] Max retries reached for container 8b5d6c8a752a. Exiting log stream.
2026-03-16 05:00:11.839 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5759/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:11.842 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:11.942 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5759/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:11.945 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-16 05:00:11.946 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-16 05:00:11.946 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5759/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:11.949 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.186.175/tcp/5760/p2p/16Uiu2HAm4tGPH1jvov5TsYkFiwW6tDRV67kGGD6GBvcqcNrBRkk9","/ip4/172.18.186.175/tcp/5761/ws/p2p/16Uiu2HAm4tGPH1jvov5TsYkFiwW6tDRV67kGGD6GBvcqcNrBRkk9"],"enrUri":"enr:-L24QJB0wqpun55AiJ4TZjOhGBZm3ww1hJ3-KclU2hzAOB4RXZZ7PZ3ejZKLjkVoSDBMNYbShpWttt4Y_DfN7vXHPGACgmlkgnY0gmlwhKwSuq-KbXVsdGlhZGRyc5YACASsErqvBhaAAAoErBK6rwYWgd0DgnJzhQACAQAAiXNlY3AyNTZrMaECjIkvYtHKdTdN2RsKktVQm4sFIJfVlRX2QIlVSz-hHZKDdGNwghaAg3VkcIIWgoV3YWt1MgU"}'
2026-03-16 05:00:11.949 INFO [src.node.waku_node] REST service is ready !!
2026-03-16 05:00:11.956 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-16 05:00:11.957 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:11.957 DEBUG [src.node.waku_node] Starting Node...
2026-03-16 05:00:11.957 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-16 05:00:11.958 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-16 05:00:11.959 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.121.4
2026-03-16 05:00:11.959 DEBUG [src.node.docker_mananger] Generated ports ['18685', '18686', '18687', '18688', '18689']
2026-03-16 05:00:11.959 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-16 05:00:11.959 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-16 05:00:11.959 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-16 05:00:11.959 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-16 05:00:11.960 DEBUG [src.node.waku_node] Using volumes []
2026-03-16 05:00:11.960 DEBUG [src.node.docker_mananger] docker run -i -t -p 18685:18685 -p 18686:18686 -p 18687:18687 -p 18688:18688 -p 18689:18689 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=18687 --rest-port=18685 --tcp-port=18686 --discv5-udp-port=18688 --rest-address=0.0.0.0 --nat=extip:172.18.121.4 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=9c8dda2b2b737e3d05e0ebec2cf899186db1c1ff98bcea58a1fac947f549d2c8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=18689 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJB0wqpun55AiJ4TZjOhGBZm3ww1hJ3-KclU2hzAOB4RXZZ7PZ3ejZKLjkVoSDBMNYbShpWttt4Y_DfN7vXHPGACgmlkgnY0gmlwhKwSuq-KbXVsdGlhZGRyc5YACASsErqvBhaAAAoErBK6rwYWgd0DgnJzhQACAQAAiXNlY3AyNTZrMaECjIkvYtHKdTdN2RsKktVQm4sFIJfVlRX2QIlVSz-hHZKDdGNwghaAg3VkcIIWgoV3YWt1MgU
2026-03-16 05:00:12.176 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.121.4 waku 2f32ac80886996494a5a52802f191aac306ab7b447bb1db3794a4ab88e32abe6
2026-03-16 05:00:12.209 DEBUG [src.node.docker_mananger] Container started with ID 2f32ac808869. Setting up logs at ./log/docker/node2_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:12.210 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 18685
2026-03-16 05:00:12.210 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-16 05:00:13.210 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.213 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:13.313 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.316 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:13.416 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.419 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:13.519 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.522 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:13.623 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.641 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:13.741 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.744 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-16 05:00:13.744 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-16 05:00:13.744 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:13.747 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.121.4/tcp/18686/p2p/16Uiu2HAmSzKt3z3yceQ7YHKUNMbppcnsNAeGZELCZG3pPeje4o2p","/ip4/172.18.121.4/tcp/18687/ws/p2p/16Uiu2HAmSzKt3z3yceQ7YHKUNMbppcnsNAeGZELCZG3pPeje4o2p"],"enrUri":"enr:-L24QKoHgQkuBSMLigBDmu94J_65FLSLapD6n98sXir9iMSVbNinBu0XbvSCNSP213lqwgwHQASmZQD0WlCHj3SsXOECgmlkgnY0gmlwhKwSeQSKbXVsdGlhZGRyc5YACASsEnkEBkj-AAoErBJ5BAZI_90DgnJzhQACAQAAiXNlY3AyNTZrMaED1Pe8RG0nWm7mEe2DlnAuqEkL3FgXkgLnQedjbNQkOCmDdGNwgkj-g3VkcIJJAIV3YWt1MgE"}'
2026-03-16 05:00:13.747 INFO [src.node.waku_node] REST service is ready !!
2026-03-16 05:00:13.747 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18685/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.186.175/tcp/5760/p2p/16Uiu2HAm4tGPH1jvov5TsYkFiwW6tDRV67kGGD6GBvcqcNrBRkk9"]'
2026-03-16 05:00:13.769 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:13.783 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-16 05:00:13.784 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:13.784 DEBUG [src.node.waku_node] Starting Node...
2026-03-16 05:00:13.784 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-16 05:00:13.786 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-16 05:00:13.787 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.95.12
2026-03-16 05:00:13.788 DEBUG [src.node.docker_mananger] Generated ports ['63023', '63024', '63025', '63026', '63027']
2026-03-16 05:00:13.789 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-16 05:00:13.790 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-16 05:00:13.790 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-16 05:00:13.790 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-16 05:00:13.790 DEBUG [src.node.waku_node] Using volumes []
2026-03-16 05:00:13.790 DEBUG [src.node.docker_mananger] docker run -i -t -p 63023:63023 -p 63024:63024 -p 63025:63025 -p 63026:63026 -p 63027:63027 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=63025 --rest-port=63023 --tcp-port=63024 --discv5-udp-port=63026 --rest-address=0.0.0.0 --nat=extip:172.18.95.12 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=94a7bfddebf8f6f6301ba57b3f7dcbdaddfb0ac06ea250dbf8c54f6e5df7ffbf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=63027 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJB0wqpun55AiJ4TZjOhGBZm3ww1hJ3-KclU2hzAOB4RXZZ7PZ3ejZKLjkVoSDBMNYbShpWttt4Y_DfN7vXHPGACgmlkgnY0gmlwhKwSuq-KbXVsdGlhZGRyc5YACASsErqvBhaAAAoErBK6rwYWgd0DgnJzhQACAQAAiXNlY3AyNTZrMaECjIkvYtHKdTdN2RsKktVQm4sFIJfVlRX2QIlVSz-hHZKDdGNwghaAg3VkcIIWgoV3YWt1MgU
2026-03-16 05:00:13.996 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.95.12 waku c7e774782d0499b9f8a188d49d36b11c2baa6ce5e262031b0545855978de24ee
2026-03-16 05:00:14.035 DEBUG [src.node.docker_mananger] Container started with ID c7e774782d04. Setting up logs at ./log/docker/node3_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:14.035 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 63023
2026-03-16 05:00:14.035 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-16 05:00:15.036 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63023/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:15.039 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:15.139 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63023/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:15.142 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-16 05:00:15.142 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-16 05:00:15.143 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63023/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:15.145 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.95.12/tcp/63024/p2p/16Uiu2HAkyb7KE17jGrd6jJDiZ9Pae92t9d82tN4cvdiNv5AVUYn4","/ip4/172.18.95.12/tcp/63025/ws/p2p/16Uiu2HAkyb7KE17jGrd6jJDiZ9Pae92t9d82tN4cvdiNv5AVUYn4"],"enrUri":"enr:-L24QNL5oXJr6394CewXTMFh1yYu-QCFXNVHEIkHrZCj2WeZX2xLicigYtpUSjaIqaTiU4rzikMGSHNS9-7wu6UMrswCgmlkgnY0gmlwhKwSXwyKbXVsdGlhZGRyc5YACASsEl8MBvYwAAoErBJfDAb2Md0DgnJzhQACAQAAiXNlY3AyNTZrMaECPdmfE4Umo4EFxa45OdFshk6a3kP9q-bxu9qxbIT0oKmDdGNwgvYwg3VkcIL2MoV3YWt1MgE"}'
2026-03-16 05:00:15.145 INFO [src.node.waku_node] REST service is ready !!
2026-03-16 05:00:15.146 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63023/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.186.175/tcp/5760/p2p/16Uiu2HAm4tGPH1jvov5TsYkFiwW6tDRV67kGGD6GBvcqcNrBRkk9"]'
2026-03-16 05:00:15.181 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:15.194 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-16 05:00:15.195 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:15.199 DEBUG [src.node.waku_node] Starting Node...
2026-03-16 05:00:15.199 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-16 05:00:15.202 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-16 05:00:15.204 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.61.242
2026-03-16 05:00:15.204 DEBUG [src.node.docker_mananger] Generated ports ['16784', '16785', '16786', '16787', '16788']
2026-03-16 05:00:15.205 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-16 05:00:15.205 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-16 05:00:15.206 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-16 05:00:15.208 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-16 05:00:15.209 DEBUG [src.node.waku_node] Using volumes []
2026-03-16 05:00:15.209 DEBUG [src.node.docker_mananger] docker run -i -t -p 16784:16784 -p 16785:16785 -p 16786:16786 -p 16787:16787 -p 16788:16788 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=16786 --rest-port=16784 --tcp-port=16785 --discv5-udp-port=16787 --rest-address=0.0.0.0 --nat=extip:172.18.61.242 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=c126fc85ec47a1cd9adafeda816bb7c3157c649fbfa40fbea29e999df5eea0ed --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=16788 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJB0wqpun55AiJ4TZjOhGBZm3ww1hJ3-KclU2hzAOB4RXZZ7PZ3ejZKLjkVoSDBMNYbShpWttt4Y_DfN7vXHPGACgmlkgnY0gmlwhKwSuq-KbXVsdGlhZGRyc5YACASsErqvBhaAAAoErBK6rwYWgd0DgnJzhQACAQAAiXNlY3AyNTZrMaECjIkvYtHKdTdN2RsKktVQm4sFIJfVlRX2QIlVSz-hHZKDdGNwghaAg3VkcIIWgoV3YWt1MgU
2026-03-16 05:00:15.414 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.61.242 waku 28d966f15a1549df96c7f753f728db5802490a18b892a9cc93190523921978f7
2026-03-16 05:00:15.452 DEBUG [src.node.docker_mananger] Container started with ID 28d966f15a15. Setting up logs at ./log/docker/node4_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:15.452 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 16784
2026-03-16 05:00:15.453 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-16 05:00:16.454 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16784/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:16.456 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:16.557 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16784/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:16.559 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-16 05:00:16.559 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-16 05:00:16.560 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16784/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:16.563 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.61.242/tcp/16785/p2p/16Uiu2HAkvaQ84LgJR4HkrVyUXpq8Mmb1UyWqnCCHqxf9Fn2VL8S1","/ip4/172.18.61.242/tcp/16786/ws/p2p/16Uiu2HAkvaQ84LgJR4HkrVyUXpq8Mmb1UyWqnCCHqxf9Fn2VL8S1"],"enrUri":"enr:-L24QMTHBqbkRJIda3L4aUV7zC5AVTg7XpEEQgzfRPAcxHUdHQX5Xir9ajv1OUeGYjWUHwUVaZCyvCXJiPEY_tb3h6gCgmlkgnY0gmlwhKwSPfKKbXVsdGlhZGRyc5YACASsEj3yBkGRAAoErBI98gZBkt0DgnJzhQACAQAAiXNlY3AyNTZrMaECERf77Dedjbv_phNoVUhDirTuXEspdKGDboU0OzN2Ij6DdGNwgkGRg3VkcIJBk4V3YWt1MgE"}'
2026-03-16 05:00:16.563 INFO [src.node.waku_node] REST service is ready !!
2026-03-16 05:00:16.564 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16784/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.186.175/tcp/5760/p2p/16Uiu2HAm4tGPH1jvov5TsYkFiwW6tDRV67kGGD6GBvcqcNrBRkk9"]'
2026-03-16 05:00:16.602 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:16.622 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-16 05:00:16.622 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:16.622 DEBUG [src.node.waku_node] Starting Node...
2026-03-16 05:00:16.623 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-16 05:00:16.627 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-16 05:00:16.627 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.39.11
2026-03-16 05:00:16.627 DEBUG [src.node.docker_mananger] Generated ports ['48272', '48273', '48274', '48275', '48276']
2026-03-16 05:00:16.628 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-16 05:00:16.628 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-16 05:00:16.629 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-16 05:00:16.629 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-16 05:00:16.630 DEBUG [src.node.waku_node] Using volumes []
2026-03-16 05:00:16.630 DEBUG [src.node.docker_mananger] docker run -i -t -p 48272:48272 -p 48273:48273 -p 48274:48274 -p 48275:48275 -p 48276:48276 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=48274 --rest-port=48272 --tcp-port=48273 --discv5-udp-port=48275 --rest-address=0.0.0.0 --nat=extip:172.18.39.11 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=2cc8cd43cfaa6bba70b38ee51c7ed076bfd7c4df17a9c3238edbe8486fcb48ed --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=48276 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJB0wqpun55AiJ4TZjOhGBZm3ww1hJ3-KclU2hzAOB4RXZZ7PZ3ejZKLjkVoSDBMNYbShpWttt4Y_DfN7vXHPGACgmlkgnY0gmlwhKwSuq-KbXVsdGlhZGRyc5YACASsErqvBhaAAAoErBK6rwYWgd0DgnJzhQACAQAAiXNlY3AyNTZrMaECjIkvYtHKdTdN2RsKktVQm4sFIJfVlRX2QIlVSz-hHZKDdGNwghaAg3VkcIIWgoV3YWt1MgU
2026-03-16 05:00:16.844 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.39.11 waku be494af72f6cbda7023eeeb0c860f26b55e0ebf2f2f1bfd9c5dbc43164a865c7
2026-03-16 05:00:16.881 DEBUG [src.node.docker_mananger] Container started with ID be494af72f6c. Setting up logs at ./log/docker/node5_2026-03-16_05-00-10__c1e773c8-2113-4ce9-90fc-cbc560fe3bb6__wakuorg_nwaku:latest.log
2026-03-16 05:00:16.881 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 48272
2026-03-16 05:00:16.882 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-16 05:00:17.882 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48272/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:17.885 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-16 05:00:17.985 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48272/health" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:17.988 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Connected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-16 05:00:17.988 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-16 05:00:17.988 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48272/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:17.991 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.39.11/tcp/48273/p2p/16Uiu2HAmC7qTC1eSCkQJ61TfcYTZKQhhn71bqteMnAn4HDgwyPQP","/ip4/172.18.39.11/tcp/48274/ws/p2p/16Uiu2HAmC7qTC1eSCkQJ61TfcYTZKQhhn71bqteMnAn4HDgwyPQP"],"enrUri":"enr:-L24QKyvhSkbBeCa-skdJ7CJjFMkob0kKfB8A0GkVAxrR1oFB-0ekKZPvJ6MfHqv75w3Xp8wvB6QSJMz7laPknj8GfECgmlkgnY0gmlwhKwSJwuKbXVsdGlhZGRyc5YACASsEicLBryRAAoErBInCwa8kt0DgnJzhQACAQAAiXNlY3AyNTZrMaEC-ATvAdBnaeRaZpzBOA-ap92byK91QPbSv460iaVUP-SDdGNwgryRg3VkcIK8k4V3YWt1MgE"}'
2026-03-16 05:00:17.991 INFO [src.node.waku_node] REST service is ready !!
2026-03-16 05:00:17.991 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48272/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.186.175/tcp/5760/p2p/16Uiu2HAm4tGPH1jvov5TsYkFiwW6tDRV67kGGD6GBvcqcNrBRkk9"]'
2026-03-16 05:00:18.028 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.031 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:5759/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-16 05:00:18.039 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.041 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:18685/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-16 05:00:18.049 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.050 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:63023/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-16 05:00:18.055 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.056 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16784/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-16 05:00:18.062 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.062 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48272/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-16 05:00:18.067 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.069 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:5759/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-16 05:00:18.087 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-16 05:00:18.088 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-16 05:00:18.190 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-16 05:00:18.192 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5759/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:18.199 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773637218069317958,"ephemeral":false,"proof":""}]'
2026-03-16 05:00:18.201 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-16 05:00:18.201 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:18685/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:18.204 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773637218069317958,"ephemeral":false,"proof":""}]'
2026-03-16 05:00:18.205 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-16 05:00:18.205 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:63023/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:18.208 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773637218069317958,"ephemeral":false,"proof":""}]'
2026-03-16 05:00:18.209 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-16 05:00:18.209 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16784/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:18.212 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773637218069317958,"ephemeral":false,"proof":""}]'
2026-03-16 05:00:18.213 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-16 05:00:18.213 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48272/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-16 05:00:18.216 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1773637218069317958,"ephemeral":false,"proof":""}]'
2026-03-16 05:00:18.219 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-16 05:00:18.220 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-16 05:00:18.220 DEBUG [src.node.waku_node] Stopping container with id f13504672fe3
2026-03-16 05:00:18.846 DEBUG [src.node.waku_node] Container stopped.
2026-03-16 05:00:18.848 DEBUG [src.node.waku_node] Stopping container with id 2f32ac808869
2026-03-16 05:00:19.482 DEBUG [src.node.waku_node] Container stopped.
2026-03-16 05:00:19.484 DEBUG [src.node.waku_node] Stopping container with id c7e774782d04
2026-03-16 05:00:19.662 ERROR [src.node.docker_mananger] Max retries reached for container f13504672fe3. Exiting log stream.
2026-03-16 05:00:20.092 DEBUG [src.node.waku_node] Container stopped.
2026-03-16 05:00:20.092 DEBUG [src.node.waku_node] Stopping container with id 28d966f15a15
2026-03-16 05:00:20.302 ERROR [src.node.docker_mananger] Max retries reached for container 2f32ac808869. Exiting log stream.
2026-03-16 05:00:20.667 DEBUG [src.node.waku_node] Container stopped.
2026-03-16 05:00:20.669 DEBUG [src.node.waku_node] Stopping container with id be494af72f6c
2026-03-16 05:00:20.911 ERROR [src.node.docker_mananger] Max retries reached for container c7e774782d04. Exiting log stream.
2026-03-16 05:00:21.266 DEBUG [src.node.waku_node] Container stopped.
2026-03-16 05:00:21.270 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-16 05:00:21.321 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-16 05:00:21.352 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-16 05:00:21.387 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-16 05:00:21.422 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-16 05:00:21.457 DEBUG [src.node.docker_mananger] No errors found in the waku logs.