192 lines
33 KiB
Plaintext

2026-03-26 04:55:22.089 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-26 04:55:22.089 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28
2026-03-26 04:55:22.090 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-26 04:55:22.090 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-26 04:55:22.090 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-26 04:55:22.097 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:55:22.098 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:22.098 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:55:22.098 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:55:22.099 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:55:22.100 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.30.8
2026-03-26 04:55:22.100 DEBUG [src.node.docker_mananger] Generated ports ['39106', '39107', '39108', '39109', '39110']
2026-03-26 04:55:22.100 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-26 04:55:22.100 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-26 04:55:22.100 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:55:22.100 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:55:22.101 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:55:22.101 DEBUG [src.node.docker_mananger] docker run -i -t -p 39106:39106 -p 39107:39107 -p 39108:39108 -p 39109:39109 -p 39110:39110 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=39108 --rest-port=39106 --tcp-port=39107 --discv5-udp-port=39109 --rest-address=0.0.0.0 --nat=extip:172.18.30.8 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=672aa2c6dce4eef8625f5bd8edaabfafa3cc8dcbf7c9bdc4a2ab2c80f3c92d26 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=39110 --metrics-logging=true --relay=true --filter=true
2026-03-26 04:55:22.299 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.30.8 waku c181986994274339f203eaae664584ebc7277df16c8c60ca4d0ea2e317792245
2026-03-26 04:55:22.324 ERROR [src.node.docker_mananger] Max retries reached for container 9452c5eb34c3. Exiting log stream.
2026-03-26 04:55:22.335 DEBUG [src.node.docker_mananger] Container started with ID c18198699427. Setting up logs at ./log/docker/node1_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:22.336 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 39106
2026-03-26 04:55:22.336 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:55:22.892 ERROR [src.node.docker_mananger] Max retries reached for container 547c5a9a6fb1. Exiting log stream.
2026-03-26 04:55:23.337 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.340 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-26 04:55:23.440 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.442 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-26 04:55:23.543 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.545 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-26 04:55:23.646 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.648 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-26 04:55:23.748 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.751 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-26 04:55:23.851 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.854 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"INITIALIZING","connectionStatus":"Disconnected","protocolsHealth":[]}'
2026-03-26 04:55:23.954 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.957 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"READY"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:55:23.957 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:55:23.957 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:23.960 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.30.8/tcp/39107/p2p/16Uiu2HAmBHVLdYVsTH8zvDtJUucFaMRZnLfJkAt8un6mB61QRpcm","/ip4/172.18.30.8/tcp/39108/ws/p2p/16Uiu2HAmBHVLdYVsTH8zvDtJUucFaMRZnLfJkAt8un6mB61QRpcm"],"enrUri":"enr:-L24QJSYPODly13HCF8fMbRd0Dh8aIXMXPoyU6y9EnKXoOM1TqWX5p1ajyos-ZbS7YjoOtHlDpYGbpLStg0A79u04osCgmlkgnY0gmlwhKwSHgiKbXVsdGlhZGRyc5YACASsEh4IBpjDAAoErBIeCAaYxN0DgnJzhQACAQAAiXNlY3AyNTZrMaEC66JQFbXG75GF_0Auq6HNOxs2DvOgVgb4fN7XLfyN7CaDdGNwgpjDg3VkcIKYxYV3YWt1MgU"}'
2026-03-26 04:55:23.960 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:55:23.967 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:55:23.967 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:23.967 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:55:23.967 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:55:23.968 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:55:23.969 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.69.43
2026-03-26 04:55:23.969 DEBUG [src.node.docker_mananger] Generated ports ['25271', '25272', '25273', '25274', '25275']
2026-03-26 04:55:23.969 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-26 04:55:23.969 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-26 04:55:23.969 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:55:23.969 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:55:23.969 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:55:23.969 DEBUG [src.node.docker_mananger] docker run -i -t -p 25271:25271 -p 25272:25272 -p 25273:25273 -p 25274:25274 -p 25275:25275 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=25273 --rest-port=25271 --tcp-port=25272 --discv5-udp-port=25274 --rest-address=0.0.0.0 --nat=extip:172.18.69.43 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=38e0ea16aabbc8ce90ac8ed147bdbc0cb2e7102b6c313a3a17a10632baae254b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=25275 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJSYPODly13HCF8fMbRd0Dh8aIXMXPoyU6y9EnKXoOM1TqWX5p1ajyos-ZbS7YjoOtHlDpYGbpLStg0A79u04osCgmlkgnY0gmlwhKwSHgiKbXVsdGlhZGRyc5YACASsEh4IBpjDAAoErBIeCAaYxN0DgnJzhQACAQAAiXNlY3AyNTZrMaEC66JQFbXG75GF_0Auq6HNOxs2DvOgVgb4fN7XLfyN7CaDdGNwgpjDg3VkcIKYxYV3YWt1MgU
2026-03-26 04:55:24.182 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.69.43 waku 08c76e6c34a829f9de007a67d6642701a740d206f93d5f930ded146279374af6
2026-03-26 04:55:24.218 DEBUG [src.node.docker_mananger] Container started with ID 08c76e6c34a8. Setting up logs at ./log/docker/node2_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:24.218 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 25271
2026-03-26 04:55:24.218 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:55:25.219 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25271/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:25.221 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:55:25.222 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:55:25.222 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25271/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:25.224 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.69.43/tcp/25272/p2p/16Uiu2HAm8joN5gDjK1pfUPctE3A3xXngqDScrCJoMZeSzXs66mFp","/ip4/172.18.69.43/tcp/25273/ws/p2p/16Uiu2HAm8joN5gDjK1pfUPctE3A3xXngqDScrCJoMZeSzXs66mFp"],"enrUri":"enr:-L24QLdBkkOrU-m2RYzdNaqziO86EP-adSVgl4oE3-L4w2FDIWr1enqqKI7m_CdpHkESNDLfwZs-bCUbsKF3X4KXu4UCgmlkgnY0gmlwhKwSRSuKbXVsdGlhZGRyc5YACASsEkUrBmK4AAoErBJFKwZiud0DgnJzhQACAQAAiXNlY3AyNTZrMaECxcy76YGlgpfllQdzJFGh8wmNVgLG294nuYeNPFKzL-ODdGNwgmK4g3VkcIJiuoV3YWt1MgE"}'
2026-03-26 04:55:25.225 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:55:25.225 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25271/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.30.8/tcp/39107/p2p/16Uiu2HAmBHVLdYVsTH8zvDtJUucFaMRZnLfJkAt8un6mB61QRpcm"]'
2026-03-26 04:55:25.250 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:25.272 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:55:25.273 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:25.274 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:55:25.274 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:55:25.277 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:55:25.278 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.49.46
2026-03-26 04:55:25.279 DEBUG [src.node.docker_mananger] Generated ports ['36443', '36444', '36445', '36446', '36447']
2026-03-26 04:55:25.280 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-26 04:55:25.282 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-26 04:55:25.283 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:55:25.284 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:55:25.285 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:55:25.287 DEBUG [src.node.docker_mananger] docker run -i -t -p 36443:36443 -p 36444:36444 -p 36445:36445 -p 36446:36446 -p 36447:36447 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=36445 --rest-port=36443 --tcp-port=36444 --discv5-udp-port=36446 --rest-address=0.0.0.0 --nat=extip:172.18.49.46 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=edb7b50b2583019ddb6c57b100a3b4149a975da3bef2f5bdbaac135d89dc8dff --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=36447 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJSYPODly13HCF8fMbRd0Dh8aIXMXPoyU6y9EnKXoOM1TqWX5p1ajyos-ZbS7YjoOtHlDpYGbpLStg0A79u04osCgmlkgnY0gmlwhKwSHgiKbXVsdGlhZGRyc5YACASsEh4IBpjDAAoErBIeCAaYxN0DgnJzhQACAQAAiXNlY3AyNTZrMaEC66JQFbXG75GF_0Auq6HNOxs2DvOgVgb4fN7XLfyN7CaDdGNwgpjDg3VkcIKYxYV3YWt1MgU
2026-03-26 04:55:25.514 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.49.46 waku b7aefd00acdbfa90a228bcdcc5b8ba0776919429362945fbfbce6c00f9dec626
2026-03-26 04:55:25.551 DEBUG [src.node.docker_mananger] Container started with ID b7aefd00acdb. Setting up logs at ./log/docker/node3_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:25.551 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 36443
2026-03-26 04:55:25.551 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:55:26.552 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36443/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:26.554 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:55:26.555 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:55:26.555 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36443/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:26.557 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.49.46/tcp/36444/p2p/16Uiu2HAkvb1azcrfeZtq8viPMb5iXLPLtwsD8FCbQcsEXbGqZW3d","/ip4/172.18.49.46/tcp/36445/ws/p2p/16Uiu2HAkvb1azcrfeZtq8viPMb5iXLPLtwsD8FCbQcsEXbGqZW3d"],"enrUri":"enr:-L24QFlDD_oKKn5Vsr295jJ8IvjeZNTfx2pMF3XlR7kXypeeKUOekzvi8dc6WEZi60SjT1CogUGhUSAkbQjA0odOTYUCgmlkgnY0gmlwhKwSMS6KbXVsdGlhZGRyc5YACASsEjEuBo5cAAoErBIxLgaOXd0DgnJzhQACAQAAiXNlY3AyNTZrMaECEUAVdJy5eU7y8vJuyLeFzcw-pbZko4Qq5AuiBR57boyDdGNwgo5cg3VkcIKOXoV3YWt1MgE"}'
2026-03-26 04:55:26.557 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:55:26.558 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:36443/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.30.8/tcp/39107/p2p/16Uiu2HAmBHVLdYVsTH8zvDtJUucFaMRZnLfJkAt8un6mB61QRpcm"]'
2026-03-26 04:55:26.581 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:26.600 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:55:26.602 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:26.602 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:55:26.603 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:55:26.606 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:55:26.607 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.63.130
2026-03-26 04:55:26.608 DEBUG [src.node.docker_mananger] Generated ports ['48480', '48481', '48482', '48483', '48484']
2026-03-26 04:55:26.611 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-26 04:55:26.611 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-26 04:55:26.611 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:55:26.612 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:55:26.613 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:55:26.613 DEBUG [src.node.docker_mananger] docker run -i -t -p 48480:48480 -p 48481:48481 -p 48482:48482 -p 48483:48483 -p 48484:48484 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=48482 --rest-port=48480 --tcp-port=48481 --discv5-udp-port=48483 --rest-address=0.0.0.0 --nat=extip:172.18.63.130 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=edef4aad33ef9f34dd50019c25865bdd65b5be4bfcbc5db0dc980a90de6479ae --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=48484 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJSYPODly13HCF8fMbRd0Dh8aIXMXPoyU6y9EnKXoOM1TqWX5p1ajyos-ZbS7YjoOtHlDpYGbpLStg0A79u04osCgmlkgnY0gmlwhKwSHgiKbXVsdGlhZGRyc5YACASsEh4IBpjDAAoErBIeCAaYxN0DgnJzhQACAQAAiXNlY3AyNTZrMaEC66JQFbXG75GF_0Auq6HNOxs2DvOgVgb4fN7XLfyN7CaDdGNwgpjDg3VkcIKYxYV3YWt1MgU
2026-03-26 04:55:26.832 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.63.130 waku 2f972077b241210c32245fb92febad6c4ced37508f68765ef502d9470ad2de14
2026-03-26 04:55:26.868 DEBUG [src.node.docker_mananger] Container started with ID 2f972077b241. Setting up logs at ./log/docker/node4_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:26.868 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 48480
2026-03-26 04:55:26.868 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:55:27.869 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48480/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:27.872 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:55:27.872 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:55:27.873 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48480/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:27.875 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.63.130/tcp/48481/p2p/16Uiu2HAky2EYu1igss8eQinZrQ5Vc4XFdPQXGB12BYvmn2PUeAan","/ip4/172.18.63.130/tcp/48482/ws/p2p/16Uiu2HAky2EYu1igss8eQinZrQ5Vc4XFdPQXGB12BYvmn2PUeAan"],"enrUri":"enr:-L24QOVi80mUjKUfcQFsk3lsQyfuJHkIrqnK-UOjdsw4Q3x_QvjaLc7GJ6mBo0ANqkiK0cwnBJMylyGk82PekBPW-vUCgmlkgnY0gmlwhKwSP4KKbXVsdGlhZGRyc5YACASsEj-CBr1hAAoErBI_gga9Yt0DgnJzhQACAQAAiXNlY3AyNTZrMaECNW2i25TxiU6_pMDqHeiFvcCjm75J-Z-iJg7PpOP0CuODdGNwgr1hg3VkcIK9Y4V3YWt1MgE"}'
2026-03-26 04:55:27.875 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:55:27.875 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48480/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.30.8/tcp/39107/p2p/16Uiu2HAmBHVLdYVsTH8zvDtJUucFaMRZnLfJkAt8un6mB61QRpcm"]'
2026-03-26 04:55:27.910 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:27.926 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:55:27.926 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:27.926 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:55:27.926 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:55:27.927 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:55:27.928 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.255.104
2026-03-26 04:55:27.928 DEBUG [src.node.docker_mananger] Generated ports ['62792', '62793', '62794', '62795', '62796']
2026-03-26 04:55:27.928 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-26 04:55:27.928 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-26 04:55:27.928 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:55:27.928 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:55:27.928 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:55:27.929 DEBUG [src.node.docker_mananger] docker run -i -t -p 62792:62792 -p 62793:62793 -p 62794:62794 -p 62795:62795 -p 62796:62796 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62794 --rest-port=62792 --tcp-port=62793 --discv5-udp-port=62795 --rest-address=0.0.0.0 --nat=extip:172.18.255.104 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=607f51b134e99053f6856ff30fed2eb6d67c9df94efdcbfde615ebefd15eeff3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62796 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QJSYPODly13HCF8fMbRd0Dh8aIXMXPoyU6y9EnKXoOM1TqWX5p1ajyos-ZbS7YjoOtHlDpYGbpLStg0A79u04osCgmlkgnY0gmlwhKwSHgiKbXVsdGlhZGRyc5YACASsEh4IBpjDAAoErBIeCAaYxN0DgnJzhQACAQAAiXNlY3AyNTZrMaEC66JQFbXG75GF_0Auq6HNOxs2DvOgVgb4fN7XLfyN7CaDdGNwgpjDg3VkcIKYxYV3YWt1MgU
2026-03-26 04:55:28.134 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.255.104 waku c265adfdc8b3c99e2287b10e9b44db6ef3873c6e3932d37b6f9aedfe57e1c630
2026-03-26 04:55:28.188 DEBUG [src.node.docker_mananger] Container started with ID c265adfdc8b3. Setting up logs at ./log/docker/node5_2026-03-26_04-55-22__f919eaf2-a2a6-4bf7-a4c0-238091563d28__wakuorg_nwaku:latest.log
2026-03-26 04:55:28.188 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 62792
2026-03-26 04:55:28.189 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:55:29.190 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62792/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.194 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"PartiallyConnected","protocolsHealth":[{"Relay":"READY"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:55:29.195 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:55:29.195 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62792/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.197 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.255.104/tcp/62793/p2p/16Uiu2HAmQUuVWvvEe1fAyi9w48kYZFXNajExPiRi3zbNBo9TvkBd","/ip4/172.18.255.104/tcp/62794/ws/p2p/16Uiu2HAmQUuVWvvEe1fAyi9w48kYZFXNajExPiRi3zbNBo9TvkBd"],"enrUri":"enr:-L24QOTyEc7jR1PktTiMuagF57omQibVpq3wt7NaZTonpHkBR2K6Pdt8DKOPD4Js0MbOEyXjafF7shj_9oFLHNYf6oYCgmlkgnY0gmlwhKwS_2iKbXVsdGlhZGRyc5YACASsEv9oBvVJAAoErBL_aAb1St0DgnJzhQACAQAAiXNlY3AyNTZrMaEDr7bxH8mtl-GyMQXClN1CcpJIz8ppVVRLWl6niA7219yDdGNwgvVJg3VkcIL1S4V3YWt1MgE"}'
2026-03-26 04:55:29.197 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:55:29.198 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62792/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.30.8/tcp/39107/p2p/16Uiu2HAmBHVLdYVsTH8zvDtJUucFaMRZnLfJkAt8un6mB61QRpcm"]'
2026-03-26 04:55:29.217 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.219 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39106/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-26 04:55:29.231 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.233 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25271/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-26 04:55:29.241 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.243 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:36443/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-26 04:55:29.247 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.248 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48480/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-26 04:55:29.255 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.256 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62792/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-26 04:55:29.261 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.261 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39106/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:55:29.275 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:55:29.276 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-26 04:55:29.376 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-26 04:55:29.377 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39106/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.382 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774500929261557586,"ephemeral":false,"proof":""}]'
2026-03-26 04:55:29.384 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-26 04:55:29.385 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25271/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.390 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774500929261557586,"ephemeral":false,"proof":""}]'
2026-03-26 04:55:29.391 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-26 04:55:29.392 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36443/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.394 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774500929261557586,"ephemeral":false,"proof":""}]'
2026-03-26 04:55:29.395 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-26 04:55:29.396 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48480/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.398 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774500929261557586,"ephemeral":false,"proof":""}]'
2026-03-26 04:55:29.399 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-26 04:55:29.400 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62792/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:55:29.402 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774500929261557586,"ephemeral":false,"proof":""}]'
2026-03-26 04:55:29.405 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-26 04:55:29.406 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-26 04:55:29.406 DEBUG [src.node.waku_node] Stopping container with id c18198699427
2026-03-26 04:55:30.048 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:55:30.049 DEBUG [src.node.waku_node] Stopping container with id 08c76e6c34a8
2026-03-26 04:55:30.645 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:55:30.646 DEBUG [src.node.waku_node] Stopping container with id b7aefd00acdb
2026-03-26 04:55:30.865 ERROR [src.node.docker_mananger] Max retries reached for container c18198699427. Exiting log stream.
2026-03-26 04:55:31.229 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:55:31.231 DEBUG [src.node.waku_node] Stopping container with id 2f972077b241
2026-03-26 04:55:31.464 ERROR [src.node.docker_mananger] Max retries reached for container 08c76e6c34a8. Exiting log stream.
2026-03-26 04:55:31.849 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:55:31.849 DEBUG [src.node.waku_node] Stopping container with id c265adfdc8b3
2026-03-26 04:55:32.152 ERROR [src.node.docker_mananger] Max retries reached for container b7aefd00acdb. Exiting log stream.
2026-03-26 04:55:32.447 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:55:32.450 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-26 04:55:32.490 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:55:32.516 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:55:32.559 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:55:32.580 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:55:32.599 DEBUG [src.node.docker_mananger] No errors found in the waku logs.