180 lines
32 KiB
Plaintext

2025-12-10 04:11:51.393 DEBUG [tests.conftest] Running fixture setup: test_id
2025-12-10 04:11:51.394 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628
2025-12-10 04:11:51.394 DEBUG [src.steps.common] Running fixture setup: common_setup
2025-12-10 04:11:51.394 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2025-12-10 04:11:51.394 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2025-12-10 04:11:51.401 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-10 04:11:51.401 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:51.401 DEBUG [src.node.waku_node] Starting Node...
2025-12-10 04:11:51.401 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-10 04:11:51.403 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-10 04:11:51.403 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.89.10
2025-12-10 04:11:51.403 DEBUG [src.node.docker_mananger] Generated ports ['55120', '55121', '55122', '55123', '55124']
2025-12-10 04:11:51.403 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2025-12-10 04:11:51.403 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2025-12-10 04:11:51.404 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-10 04:11:51.404 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-10 04:11:51.404 DEBUG [src.node.waku_node] Using volumes []
2025-12-10 04:11:51.404 DEBUG [src.node.docker_mananger] docker run -i -t -p 55120:55120 -p 55121:55121 -p 55122:55122 -p 55123:55123 -p 55124:55124 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=55122 --rest-port=55120 --tcp-port=55121 --discv5-udp-port=55123 --rest-address=0.0.0.0 --nat=extip:172.18.89.10 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=2cab5878fcaafa38bf1b78ddfbceb3fa34e239c464d960c301cdee1b7398e8c1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=55124 --metrics-logging=true --relay=true --filter=true
2025-12-10 04:11:51.587 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.89.10 waku 06167c5eedce7b0713d84e424892b760cf63e42f9a4a9054857346bb2aa877f3
2025-12-10 04:11:51.619 DEBUG [src.node.docker_mananger] Container started with ID 06167c5eedce. Setting up logs at ./log/docker/node1_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:51.619 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 55120
2025-12-10 04:11:51.620 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-10 04:11:51.652 ERROR [src.node.docker_mananger] Max retries reached for container 33b86ecf03b3. Exiting log stream.
2025-12-10 04:11:52.201 ERROR [src.node.docker_mananger] Max retries reached for container 673793849f07. Exiting log stream.
2025-12-10 04:11:52.620 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55120/health" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:52.623 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2025-12-10 04:11:52.623 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-10 04:11:52.624 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55120/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:52.626 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.89.10/tcp/55121/p2p/16Uiu2HAm7JjQGPB6jjkHQbX14Pc7HizusDLVcwd5t58Eaz9ZYC8s","/ip4/172.18.89.10/tcp/55122/ws/p2p/16Uiu2HAm7JjQGPB6jjkHQbX14Pc7HizusDLVcwd5t58Eaz9ZYC8s"],"enrUri":"enr:-L24QBW-stdSyR49xjOAB2HhEFEV7_yI7Rr4keMj1UMbAAIXBdVjxDwf-NvcwuWD4cYCuvxs5ePGJN3zKd9K2wGgRRkCgmlkgnY0gmlwhKwSWQqKbXVsdGlhZGRyc5YACASsElkKBtdRAAoErBJZCgbXUt0DgnJzhQACAQAAiXNlY3AyNTZrMaECsIUL8Wn1Pm6HrY3V7p8dPRIeZniW5PsrtKDCcTz7deyDdGNwgtdRg3VkcILXU4V3YWt1MgU"}'
2025-12-10 04:11:52.626 INFO [src.node.waku_node] REST service is ready !!
2025-12-10 04:11:52.632 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-10 04:11:52.632 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:52.633 DEBUG [src.node.waku_node] Starting Node...
2025-12-10 04:11:52.633 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-10 04:11:52.634 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-10 04:11:52.634 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.13.10
2025-12-10 04:11:52.634 DEBUG [src.node.docker_mananger] Generated ports ['22433', '22434', '22435', '22436', '22437']
2025-12-10 04:11:52.635 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2025-12-10 04:11:52.635 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2025-12-10 04:11:52.635 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-10 04:11:52.635 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-10 04:11:52.635 DEBUG [src.node.waku_node] Using volumes []
2025-12-10 04:11:52.635 DEBUG [src.node.docker_mananger] docker run -i -t -p 22433:22433 -p 22434:22434 -p 22435:22435 -p 22436:22436 -p 22437:22437 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22435 --rest-port=22433 --tcp-port=22434 --discv5-udp-port=22436 --rest-address=0.0.0.0 --nat=extip:172.18.13.10 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=efdde47e96351093d0f3c42aef4edffc8ccaa2643140f7b8cd02016fffc8bdcf --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22437 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QBW-stdSyR49xjOAB2HhEFEV7_yI7Rr4keMj1UMbAAIXBdVjxDwf-NvcwuWD4cYCuvxs5ePGJN3zKd9K2wGgRRkCgmlkgnY0gmlwhKwSWQqKbXVsdGlhZGRyc5YACASsElkKBtdRAAoErBJZCgbXUt0DgnJzhQACAQAAiXNlY3AyNTZrMaECsIUL8Wn1Pm6HrY3V7p8dPRIeZniW5PsrtKDCcTz7deyDdGNwgtdRg3VkcILXU4V3YWt1MgU
2025-12-10 04:11:52.825 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.13.10 waku 94a7bf770f6302cdf4231ba029a4543013000dc23d3ad1c4b4c684ccaeeaaf23
2025-12-10 04:11:52.856 DEBUG [src.node.docker_mananger] Container started with ID 94a7bf770f63. Setting up logs at ./log/docker/node2_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:52.857 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22433
2025-12-10 04:11:52.857 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-10 04:11:53.857 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22433/health" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:53.865 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2025-12-10 04:11:53.866 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-10 04:11:53.866 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22433/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:53.871 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.13.10/tcp/22434/p2p/16Uiu2HAmH3N4QyFz9Cd7gwAZ3xP8D9eRPZvVqmbMdbxjnKqeKECu","/ip4/172.18.13.10/tcp/22435/ws/p2p/16Uiu2HAmH3N4QyFz9Cd7gwAZ3xP8D9eRPZvVqmbMdbxjnKqeKECu"],"enrUri":"enr:-L24QPRvrAZ6pqBAncXWZgkCzaeOf4wIeAR5klzHFuT9EkJzSTn5NGO5jxcbFvH250LIk7pz7Bj82PHYCX58WxJJ4OgCgmlkgnY0gmlwhKwSDQqKbXVsdGlhZGRyc5YACASsEg0KBleiAAoErBINCgZXo90DgnJzhQACAQAAiXNlY3AyNTZrMaEDQSoSbh7x4yyL9enEMVTV-NP71B3BD3uZb2WhtsnxhqaDdGNwgleig3VkcIJXpIV3YWt1MgE"}'
2025-12-10 04:11:53.871 INFO [src.node.waku_node] REST service is ready !!
2025-12-10 04:11:53.872 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22433/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.89.10/tcp/55121/p2p/16Uiu2HAm7JjQGPB6jjkHQbX14Pc7HizusDLVcwd5t58Eaz9ZYC8s"]'
2025-12-10 04:11:53.876 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:53.883 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-10 04:11:53.884 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:53.884 DEBUG [src.node.waku_node] Starting Node...
2025-12-10 04:11:53.884 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-10 04:11:53.885 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-10 04:11:53.885 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.176.194
2025-12-10 04:11:53.885 DEBUG [src.node.docker_mananger] Generated ports ['44358', '44359', '44360', '44361', '44362']
2025-12-10 04:11:53.886 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2025-12-10 04:11:53.886 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2025-12-10 04:11:53.886 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-10 04:11:53.886 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-10 04:11:53.886 DEBUG [src.node.waku_node] Using volumes []
2025-12-10 04:11:53.886 DEBUG [src.node.docker_mananger] docker run -i -t -p 44358:44358 -p 44359:44359 -p 44360:44360 -p 44361:44361 -p 44362:44362 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=44360 --rest-port=44358 --tcp-port=44359 --discv5-udp-port=44361 --rest-address=0.0.0.0 --nat=extip:172.18.176.194 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=becaecdf3dfaddeddcc83fdf5b8e26efbfaf20feafee3ccfc14c2f3afa688fb1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=44362 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QBW-stdSyR49xjOAB2HhEFEV7_yI7Rr4keMj1UMbAAIXBdVjxDwf-NvcwuWD4cYCuvxs5ePGJN3zKd9K2wGgRRkCgmlkgnY0gmlwhKwSWQqKbXVsdGlhZGRyc5YACASsElkKBtdRAAoErBJZCgbXUt0DgnJzhQACAQAAiXNlY3AyNTZrMaECsIUL8Wn1Pm6HrY3V7p8dPRIeZniW5PsrtKDCcTz7deyDdGNwgtdRg3VkcILXU4V3YWt1MgU
2025-12-10 04:11:54.076 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.176.194 waku 3cd9b2d5e1a56121b57ca36ebb6ccbd988d7476225b4bd5d1aa5b58464cc04a4
2025-12-10 04:11:54.106 DEBUG [src.node.docker_mananger] Container started with ID 3cd9b2d5e1a5. Setting up logs at ./log/docker/node3_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:54.107 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 44358
2025-12-10 04:11:54.107 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-10 04:11:55.108 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44358/health" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:55.116 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2025-12-10 04:11:55.116 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-10 04:11:55.118 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44358/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:55.122 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.176.194/tcp/44359/p2p/16Uiu2HAm3mGAp4wsFH1rm6ZKbQ4ooqpPV77n9jCzA1LfkecVFvBe","/ip4/172.18.176.194/tcp/44360/ws/p2p/16Uiu2HAm3mGAp4wsFH1rm6ZKbQ4ooqpPV77n9jCzA1LfkecVFvBe"],"enrUri":"enr:-L24QPmvHmL2fWHSUTL_93h4RiZJDE0EmgmxnrYKYCD1F6IQEEinXZ87VD0cZeg_jA9VSajMNm4D9hUEjIZxqLVhFz8CgmlkgnY0gmlwhKwSsMKKbXVsdGlhZGRyc5YACASsErDCBq1HAAoErBKwwgatSN0DgnJzhQACAQAAiXNlY3AyNTZrMaECe-IwhCYfA9hTsbEmAKL5fPnPhtiF7sdRkW1jmTytIm2DdGNwgq1Hg3VkcIKtSYV3YWt1MgE"}'
2025-12-10 04:11:55.123 INFO [src.node.waku_node] REST service is ready !!
2025-12-10 04:11:55.123 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:44358/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.89.10/tcp/55121/p2p/16Uiu2HAm7JjQGPB6jjkHQbX14Pc7HizusDLVcwd5t58Eaz9ZYC8s"]'
2025-12-10 04:11:55.127 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:55.133 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-10 04:11:55.133 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:55.133 DEBUG [src.node.waku_node] Starting Node...
2025-12-10 04:11:55.133 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-10 04:11:55.135 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-10 04:11:55.135 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.184.22
2025-12-10 04:11:55.135 DEBUG [src.node.docker_mananger] Generated ports ['39918', '39919', '39920', '39921', '39922']
2025-12-10 04:11:55.135 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2025-12-10 04:11:55.135 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2025-12-10 04:11:55.135 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-10 04:11:55.136 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-10 04:11:55.136 DEBUG [src.node.waku_node] Using volumes []
2025-12-10 04:11:55.136 DEBUG [src.node.docker_mananger] docker run -i -t -p 39918:39918 -p 39919:39919 -p 39920:39920 -p 39921:39921 -p 39922:39922 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=39920 --rest-port=39918 --tcp-port=39919 --discv5-udp-port=39921 --rest-address=0.0.0.0 --nat=extip:172.18.184.22 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=1ebb6da71a119c97ff0447ed6ebf1bf01aeab568ff7eaacbde50abbaebfd0a7a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=39922 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QBW-stdSyR49xjOAB2HhEFEV7_yI7Rr4keMj1UMbAAIXBdVjxDwf-NvcwuWD4cYCuvxs5ePGJN3zKd9K2wGgRRkCgmlkgnY0gmlwhKwSWQqKbXVsdGlhZGRyc5YACASsElkKBtdRAAoErBJZCgbXUt0DgnJzhQACAQAAiXNlY3AyNTZrMaECsIUL8Wn1Pm6HrY3V7p8dPRIeZniW5PsrtKDCcTz7deyDdGNwgtdRg3VkcILXU4V3YWt1MgU
2025-12-10 04:11:55.338 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.184.22 waku 8d544c4462bdb6ade22197b16d92506741cc802038b1cdb01491327ce14c7ee4
2025-12-10 04:11:55.369 DEBUG [src.node.docker_mananger] Container started with ID 8d544c4462bd. Setting up logs at ./log/docker/node4_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:55.369 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 39918
2025-12-10 04:11:55.369 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-10 04:11:56.370 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39918/health" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:56.378 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2025-12-10 04:11:56.379 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-10 04:11:56.379 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39918/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:56.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.184.22/tcp/39919/p2p/16Uiu2HAkzK8YWWmwUXTVKB9biMwVzYXg9aAb5ps26jjA1ZUTH19p","/ip4/172.18.184.22/tcp/39920/ws/p2p/16Uiu2HAkzK8YWWmwUXTVKB9biMwVzYXg9aAb5ps26jjA1ZUTH19p"],"enrUri":"enr:-L24QO5Z8ZdDRoYsfy-3tL98gYIqTZRqbUKg9GcnodQFV09CYTyI3hV44wJKZ2x0ymQMkkNQnxzznr6M4D9b0Q6KXXcCgmlkgnY0gmlwhKwSuBaKbXVsdGlhZGRyc5YACASsErgWBpvvAAoErBK4Fgab8N0DgnJzhQACAQAAiXNlY3AyNTZrMaECSJ1nsf4m6Eym2BjepHoPYryXZnWeCzoJP-GlZGQJFP-DdGNwgpvvg3VkcIKb8YV3YWt1MgE"}'
2025-12-10 04:11:56.385 INFO [src.node.waku_node] REST service is ready !!
2025-12-10 04:11:56.385 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39918/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.89.10/tcp/55121/p2p/16Uiu2HAm7JjQGPB6jjkHQbX14Pc7HizusDLVcwd5t58Eaz9ZYC8s"]'
2025-12-10 04:11:56.388 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:56.394 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-10 04:11:56.395 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:56.395 DEBUG [src.node.waku_node] Starting Node...
2025-12-10 04:11:56.395 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-10 04:11:56.396 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-10 04:11:56.396 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.4.160
2025-12-10 04:11:56.396 DEBUG [src.node.docker_mananger] Generated ports ['3457', '3458', '3459', '3460', '3461']
2025-12-10 04:11:56.397 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2025-12-10 04:11:56.397 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2025-12-10 04:11:56.397 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-10 04:11:56.397 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-10 04:11:56.397 DEBUG [src.node.waku_node] Using volumes []
2025-12-10 04:11:56.397 DEBUG [src.node.docker_mananger] docker run -i -t -p 3457:3457 -p 3458:3458 -p 3459:3459 -p 3460:3460 -p 3461:3461 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=3459 --rest-port=3457 --tcp-port=3458 --discv5-udp-port=3460 --rest-address=0.0.0.0 --nat=extip:172.18.4.160 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=aa75ed6ebd66beadb63caad9b882584eacb5056a1eae9a4ca08def6aff0c6ec6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=3461 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QBW-stdSyR49xjOAB2HhEFEV7_yI7Rr4keMj1UMbAAIXBdVjxDwf-NvcwuWD4cYCuvxs5ePGJN3zKd9K2wGgRRkCgmlkgnY0gmlwhKwSWQqKbXVsdGlhZGRyc5YACASsElkKBtdRAAoErBJZCgbXUt0DgnJzhQACAQAAiXNlY3AyNTZrMaECsIUL8Wn1Pm6HrY3V7p8dPRIeZniW5PsrtKDCcTz7deyDdGNwgtdRg3VkcILXU4V3YWt1MgU
2025-12-10 04:11:56.582 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.4.160 waku 1db03b49b10ac302a167519054c8f1840ab906cd3e6d22ad7e7b27923430acee
2025-12-10 04:11:56.614 DEBUG [src.node.docker_mananger] Container started with ID 1db03b49b10a. Setting up logs at ./log/docker/node5_2025-12-10_04-11-51__fc741760-4679-4968-8a07-82968e99b628__wakuorg_nwaku:latest.log
2025-12-10 04:11:56.614 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 3457
2025-12-10 04:11:56.614 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-10 04:11:57.615 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3457/health" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.627 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"READY"}]}'
2025-12-10 04:11:57.627 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-10 04:11:57.628 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3457/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.633 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.4.160/tcp/3458/p2p/16Uiu2HAkzWwntC1Xb9jKseBRfYiHmc2pGiTdaKxoL3fHJ5PrAK8z","/ip4/172.18.4.160/tcp/3459/ws/p2p/16Uiu2HAkzWwntC1Xb9jKseBRfYiHmc2pGiTdaKxoL3fHJ5PrAK8z"],"enrUri":"enr:-L24QL0CKuTttty68sEKSNWtIPRU3_G1Y0FxwYAH_yreDRZcChuzgXtwfSiDQfGqkqKCf_RQFhLBKj5fdUBy-ycGFCoCgmlkgnY0gmlwhKwSBKCKbXVsdGlhZGRyc5YACASsEgSgBg2CAAoErBIEoAYNg90DgnJzhQACAQAAiXNlY3AyNTZrMaECS6Q3nUFM8Zk5yGX4E4bxKo5zuUkiSk6PlXKAR-1cDQ-DdGNwgg2Cg3VkcIINhIV3YWt1MgE"}'
2025-12-10 04:11:57.633 INFO [src.node.waku_node] REST service is ready !!
2025-12-10 04:11:57.633 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3457/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.89.10/tcp/55121/p2p/16Uiu2HAm7JjQGPB6jjkHQbX14Pc7HizusDLVcwd5t58Eaz9ZYC8s"]'
2025-12-10 04:11:57.636 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.636 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:55120/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2025-12-10 04:11:57.638 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.639 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22433/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2025-12-10 04:11:57.641 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.641 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:44358/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2025-12-10 04:11:57.644 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.644 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39918/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2025-12-10 04:11:57.646 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.647 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3457/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2025-12-10 04:11:57.649 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.649 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:55120/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2025-12-10 04:11:57.655 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-10 04:11:57.656 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2025-12-10 04:11:57.756 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2025-12-10 04:11:57.757 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55120/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.759 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1765339917649850087,"ephemeral":false,"proof":""}]'
2025-12-10 04:11:57.761 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2025-12-10 04:11:57.761 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22433/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.763 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1765339917649850087,"ephemeral":false,"proof":""}]'
2025-12-10 04:11:57.765 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2025-12-10 04:11:57.765 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44358/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.767 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1765339917649850087,"ephemeral":false,"proof":""}]'
2025-12-10 04:11:57.768 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2025-12-10 04:11:57.769 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39918/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.771 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1765339917649850087,"ephemeral":false,"proof":""}]'
2025-12-10 04:11:57.772 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2025-12-10 04:11:57.772 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3457/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2025-12-10 04:11:57.774 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1765339917649850087,"ephemeral":false,"proof":""}]'
2025-12-10 04:11:57.777 DEBUG [tests.conftest] Running fixture teardown: test_setup
2025-12-10 04:11:57.778 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2025-12-10 04:11:57.778 DEBUG [src.node.waku_node] Stopping container with id 06167c5eedce
2025-12-10 04:11:58.380 DEBUG [src.node.waku_node] Container stopped.
2025-12-10 04:11:58.382 DEBUG [src.node.waku_node] Stopping container with id 94a7bf770f63
2025-12-10 04:11:58.902 DEBUG [src.node.waku_node] Container stopped.
2025-12-10 04:11:58.903 DEBUG [src.node.waku_node] Stopping container with id 3cd9b2d5e1a5
2025-12-10 04:11:59.193 ERROR [src.node.docker_mananger] Max retries reached for container 06167c5eedce. Exiting log stream.
2025-12-10 04:11:59.429 DEBUG [src.node.waku_node] Container stopped.
2025-12-10 04:11:59.429 DEBUG [src.node.waku_node] Stopping container with id 8d544c4462bd
2025-12-10 04:11:59.714 ERROR [src.node.docker_mananger] Max retries reached for container 94a7bf770f63. Exiting log stream.
2025-12-10 04:11:59.939 DEBUG [src.node.waku_node] Container stopped.
2025-12-10 04:11:59.940 DEBUG [src.node.waku_node] Stopping container with id 1db03b49b10a
2025-12-10 04:12:00.241 ERROR [src.node.docker_mananger] Max retries reached for container 3cd9b2d5e1a5. Exiting log stream.
2025-12-10 04:12:00.427 DEBUG [src.node.waku_node] Container stopped.
2025-12-10 04:12:00.428 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2025-12-10 04:12:00.442 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-10 04:12:00.447 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-10 04:12:00.452 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-10 04:12:00.457 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-10 04:12:00.461 DEBUG [src.node.docker_mananger] No errors found in the waku logs.