180 lines
32 KiB
Plaintext

2026-03-01 04:46:56.406 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-01 04:46:56.407 DEBUG [tests.conftest] Running test: test_static_shard_relay with id: 2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee
2026-03-01 04:46:56.407 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-01 04:46:56.407 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-01 04:46:56.408 DEBUG [src.steps.sharding] Running fixture setup: sharding_setup
2026-03-01 04:46:56.415 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-01 04:46:56.415 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:46:56.415 DEBUG [src.node.waku_node] Starting Node...
2026-03-01 04:46:56.415 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-01 04:46:56.417 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-01 04:46:56.417 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.169.248
2026-03-01 04:46:56.417 DEBUG [src.node.docker_mananger] Generated ports ['10638', '10639', '10640', '10641', '10642']
2026-03-01 04:46:56.417 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-01 04:46:56.417 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-01 04:46:56.418 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-01 04:46:56.418 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-01 04:46:56.418 DEBUG [src.node.waku_node] Using volumes []
2026-03-01 04:46:56.418 DEBUG [src.node.docker_mananger] docker run -i -t -p 10638:10638 -p 10639:10639 -p 10640:10640 -p 10641:10641 -p 10642:10642 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=10640 --rest-port=10638 --tcp-port=10639 --discv5-udp-port=10641 --rest-address=0.0.0.0 --nat=extip:172.18.169.248 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=5221aec5ebd2e63d9dc0862cd6f6deb5c8ad8ae644d48ae069d0aa8f560a0bef --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=10642 --metrics-logging=true --relay=true --filter=true
2026-03-01 04:46:56.606 ERROR [src.node.docker_mananger] Max retries reached for container 1d44380da8fd. Exiting log stream.
2026-03-01 04:46:56.621 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.169.248 waku 838974ce13830a600c45ac9d8881f66a0dee94d2e7f9d18fca95946901a125ff
2026-03-01 04:46:56.655 DEBUG [src.node.docker_mananger] Container started with ID 838974ce1383. Setting up logs at ./log/docker/node1_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:46:56.655 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 10638
2026-03-01 04:46:56.656 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-01 04:46:57.201 ERROR [src.node.docker_mananger] Max retries reached for container c7c77e11c774. Exiting log stream.
2026-03-01 04:46:57.656 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10638/health" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:46:57.659 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-01 04:46:57.659 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-01 04:46:57.659 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10638/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:46:57.662 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.169.248/tcp/10639/p2p/16Uiu2HAkuwT6DgBgKzZxNGUUP7LktPsk4VJMWuYa7yCb3UQx7WDf","/ip4/172.18.169.248/tcp/10640/ws/p2p/16Uiu2HAkuwT6DgBgKzZxNGUUP7LktPsk4VJMWuYa7yCb3UQx7WDf"],"enrUri":"enr:-L24QDXFUNOrCIsMle53J9X19VZOCuyopHwf9oOnfVSxSTctM-KTclgZknbA7AqgoT6SsWPmZUB5ddvnpWEsn83vh6oCgmlkgnY0gmlwhKwSqfiKbXVsdGlhZGRyc5YACASsEqn4BimPAAoErBKp-AYpkN0DgnJzhQACAQAAiXNlY3AyNTZrMaECB6DZs5XtxdIc4gh-xmpx9S3H2T5IqAMzXWqWgRtCGnKDdGNwgimPg3VkcIIpkYV3YWt1MgU"}'
2026-03-01 04:46:57.662 INFO [src.node.waku_node] REST service is ready !!
2026-03-01 04:46:57.669 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-01 04:46:57.669 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:46:57.669 DEBUG [src.node.waku_node] Starting Node...
2026-03-01 04:46:57.669 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-01 04:46:57.671 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-01 04:46:57.671 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.253.172
2026-03-01 04:46:57.671 DEBUG [src.node.docker_mananger] Generated ports ['22635', '22636', '22637', '22638', '22639']
2026-03-01 04:46:57.671 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-01 04:46:57.671 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-01 04:46:57.671 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-01 04:46:57.671 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-01 04:46:57.672 DEBUG [src.node.waku_node] Using volumes []
2026-03-01 04:46:57.672 DEBUG [src.node.docker_mananger] docker run -i -t -p 22635:22635 -p 22636:22636 -p 22637:22637 -p 22638:22638 -p 22639:22639 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22637 --rest-port=22635 --tcp-port=22636 --discv5-udp-port=22638 --rest-address=0.0.0.0 --nat=extip:172.18.253.172 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=c3ebecaa25abde2efbbfd2167bbef86efddf25ffd1b189abbd7af110f64332e0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22639 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDXFUNOrCIsMle53J9X19VZOCuyopHwf9oOnfVSxSTctM-KTclgZknbA7AqgoT6SsWPmZUB5ddvnpWEsn83vh6oCgmlkgnY0gmlwhKwSqfiKbXVsdGlhZGRyc5YACASsEqn4BimPAAoErBKp-AYpkN0DgnJzhQACAQAAiXNlY3AyNTZrMaECB6DZs5XtxdIc4gh-xmpx9S3H2T5IqAMzXWqWgRtCGnKDdGNwgimPg3VkcIIpkYV3YWt1MgU
2026-03-01 04:46:57.870 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.253.172 waku bfb4df7726539eb9c7c12a53c81ce98cb2cedb577b8c5eb06f753c729bb0f75b
2026-03-01 04:46:57.904 DEBUG [src.node.docker_mananger] Container started with ID bfb4df772653. Setting up logs at ./log/docker/node2_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:46:57.904 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22635
2026-03-01 04:46:57.905 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-01 04:46:58.906 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22635/health" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:46:58.930 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-01 04:46:58.931 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-01 04:46:58.932 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22635/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:46:58.939 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.253.172/tcp/22636/p2p/16Uiu2HAmM5Uo6tSxRBVehJnqFv5ekharnBmVVrZhAMsHemEyT8n8","/ip4/172.18.253.172/tcp/22637/ws/p2p/16Uiu2HAmM5Uo6tSxRBVehJnqFv5ekharnBmVVrZhAMsHemEyT8n8"],"enrUri":"enr:-L24QIROr8b7LeOY64fOvO8-LnP2iw-gjhnneCmroKN3tLoEVEKIH8J7S_IIAyNoTHMD5TSkk8P8edy1M8h-P0HuNqsCgmlkgnY0gmlwhKwS_ayKbXVsdGlhZGRyc5YACASsEv2sBlhsAAoErBL9rAZYbd0DgnJzhQACAQAAiXNlY3AyNTZrMaEDfSOZBYEe5qEzhUDawV5U2B8T6IaizFZwij_r62c2dmWDdGNwglhsg3VkcIJYboV3YWt1MgE"}'
2026-03-01 04:46:58.939 INFO [src.node.waku_node] REST service is ready !!
2026-03-01 04:46:58.940 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22635/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.169.248/tcp/10639/p2p/16Uiu2HAkuwT6DgBgKzZxNGUUP7LktPsk4VJMWuYa7yCb3UQx7WDf"]'
2026-03-01 04:46:58.944 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:46:58.952 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-01 04:46:58.952 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:46:58.953 DEBUG [src.node.waku_node] Starting Node...
2026-03-01 04:46:58.953 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-01 04:46:58.954 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-01 04:46:58.954 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.229.29
2026-03-01 04:46:58.954 DEBUG [src.node.docker_mananger] Generated ports ['57522', '57523', '57524', '57525', '57526']
2026-03-01 04:46:58.954 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-01 04:46:58.955 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-01 04:46:58.955 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-01 04:46:58.955 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-01 04:46:58.955 DEBUG [src.node.waku_node] Using volumes []
2026-03-01 04:46:58.955 DEBUG [src.node.docker_mananger] docker run -i -t -p 57522:57522 -p 57523:57523 -p 57524:57524 -p 57525:57525 -p 57526:57526 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=57524 --rest-port=57522 --tcp-port=57523 --discv5-udp-port=57525 --rest-address=0.0.0.0 --nat=extip:172.18.229.29 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=4c0f80aaafeedf22cf14c33c44affebc1eb3fab167f3ae6e52bfcb4d12e56fff --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=57526 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDXFUNOrCIsMle53J9X19VZOCuyopHwf9oOnfVSxSTctM-KTclgZknbA7AqgoT6SsWPmZUB5ddvnpWEsn83vh6oCgmlkgnY0gmlwhKwSqfiKbXVsdGlhZGRyc5YACASsEqn4BimPAAoErBKp-AYpkN0DgnJzhQACAQAAiXNlY3AyNTZrMaECB6DZs5XtxdIc4gh-xmpx9S3H2T5IqAMzXWqWgRtCGnKDdGNwgimPg3VkcIIpkYV3YWt1MgU
2026-03-01 04:46:59.170 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.229.29 waku ce6a2b1e1695c5a62d4589cacabb54365ebf23decce3f8614ef6cc5ce9d9da7d
2026-03-01 04:46:59.203 DEBUG [src.node.docker_mananger] Container started with ID ce6a2b1e1695. Setting up logs at ./log/docker/node3_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:46:59.204 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 57522
2026-03-01 04:46:59.204 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-01 04:47:00.204 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57522/health" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:00.227 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-01 04:47:00.229 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-01 04:47:00.229 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57522/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:00.236 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.229.29/tcp/57523/p2p/16Uiu2HAmKjWs613xK2FqtWekLjNB7or4j4QEGW4DkfMifUMKbcvC","/ip4/172.18.229.29/tcp/57524/ws/p2p/16Uiu2HAmKjWs613xK2FqtWekLjNB7or4j4QEGW4DkfMifUMKbcvC"],"enrUri":"enr:-L24QBvNfWVCLpiHH_aNzsTo5Rky9rjuSLzPmzl2bB_qEC4fIiCIOGjtOfuWsRxUKYOL6go4LNWJ6s8_3IT6EYN0MdgCgmlkgnY0gmlwhKwS5R2KbXVsdGlhZGRyc5YACASsEuUdBuCzAAoErBLlHQbgtN0DgnJzhQACAQAAiXNlY3AyNTZrMaEDaSqiO481H6QXKWyn3Kx9ToKaTEIE_eqgVpYa6Jf1hamDdGNwguCzg3VkcILgtYV3YWt1MgE"}'
2026-03-01 04:47:00.236 INFO [src.node.waku_node] REST service is ready !!
2026-03-01 04:47:00.236 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:57522/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.169.248/tcp/10639/p2p/16Uiu2HAkuwT6DgBgKzZxNGUUP7LktPsk4VJMWuYa7yCb3UQx7WDf"]'
2026-03-01 04:47:00.239 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:00.247 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-01 04:47:00.247 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:47:00.247 DEBUG [src.node.waku_node] Starting Node...
2026-03-01 04:47:00.247 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-01 04:47:00.249 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-01 04:47:00.249 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.96.190
2026-03-01 04:47:00.249 DEBUG [src.node.docker_mananger] Generated ports ['17045', '17046', '17047', '17048', '17049']
2026-03-01 04:47:00.249 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-01 04:47:00.249 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-01 04:47:00.249 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-01 04:47:00.250 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-01 04:47:00.250 DEBUG [src.node.waku_node] Using volumes []
2026-03-01 04:47:00.250 DEBUG [src.node.docker_mananger] docker run -i -t -p 17045:17045 -p 17046:17046 -p 17047:17047 -p 17048:17048 -p 17049:17049 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=17047 --rest-port=17045 --tcp-port=17046 --discv5-udp-port=17048 --rest-address=0.0.0.0 --nat=extip:172.18.96.190 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=f2c48af1dbba50a7afcfa6b94def349ea6adc0a0b82c8209a87f24d2ef9c8cde --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=17049 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDXFUNOrCIsMle53J9X19VZOCuyopHwf9oOnfVSxSTctM-KTclgZknbA7AqgoT6SsWPmZUB5ddvnpWEsn83vh6oCgmlkgnY0gmlwhKwSqfiKbXVsdGlhZGRyc5YACASsEqn4BimPAAoErBKp-AYpkN0DgnJzhQACAQAAiXNlY3AyNTZrMaECB6DZs5XtxdIc4gh-xmpx9S3H2T5IqAMzXWqWgRtCGnKDdGNwgimPg3VkcIIpkYV3YWt1MgU
2026-03-01 04:47:00.468 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.96.190 waku 5369d958cc31f572795c1edaf10482b5fca914218b696072b6e6677381839a5d
2026-03-01 04:47:00.502 DEBUG [src.node.docker_mananger] Container started with ID 5369d958cc31. Setting up logs at ./log/docker/node4_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:47:00.503 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 17045
2026-03-01 04:47:00.504 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-01 04:47:01.504 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17045/health" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:01.521 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-01 04:47:01.523 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-01 04:47:01.523 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17045/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:01.530 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.96.190/tcp/17046/p2p/16Uiu2HAkvrya6iNu9LJJEmkBT2YrYwJNmDGZ2Mqpz179y9oJ8YbY","/ip4/172.18.96.190/tcp/17047/ws/p2p/16Uiu2HAkvrya6iNu9LJJEmkBT2YrYwJNmDGZ2Mqpz179y9oJ8YbY"],"enrUri":"enr:-L24QP_t7X2zOvKaLcaM0itBXEusQHlPGbLO4LIigFnmLo1BEJABrCe6Nig7b_ooN7ocjDnFRrZABD7_Nd_VdYRtGZICgmlkgnY0gmlwhKwSYL6KbXVsdGlhZGRyc5YACASsEmC-BkKWAAoErBJgvgZCl90DgnJzhQACAQAAiXNlY3AyNTZrMaECFVcZECITa99fJxquNXJOynHwJ1HZbSmScbm71LETUReDdGNwgkKWg3VkcIJCmIV3YWt1MgE"}'
2026-03-01 04:47:01.530 INFO [src.node.waku_node] REST service is ready !!
2026-03-01 04:47:01.531 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17045/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.169.248/tcp/10639/p2p/16Uiu2HAkuwT6DgBgKzZxNGUUP7LktPsk4VJMWuYa7yCb3UQx7WDf"]'
2026-03-01 04:47:01.535 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:01.542 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-01 04:47:01.542 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:47:01.543 DEBUG [src.node.waku_node] Starting Node...
2026-03-01 04:47:01.543 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-01 04:47:01.544 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-01 04:47:01.544 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.89.116
2026-03-01 04:47:01.544 DEBUG [src.node.docker_mananger] Generated ports ['11162', '11163', '11164', '11165', '11166']
2026-03-01 04:47:01.545 DEBUG [src.node.waku_node] Cluster id was resolved to: 2
2026-03-01 04:47:01.545 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-01 04:47:01.545 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-01 04:47:01.545 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-01 04:47:01.545 DEBUG [src.node.waku_node] Using volumes []
2026-03-01 04:47:01.545 DEBUG [src.node.docker_mananger] docker run -i -t -p 11162:11162 -p 11163:11163 -p 11164:11164 -p 11165:11165 -p 11166:11166 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=11164 --rest-port=11162 --tcp-port=11163 --discv5-udp-port=11165 --rest-address=0.0.0.0 --nat=extip:172.18.89.116 --peer-exchange=true --discv5-discovery=true --cluster-id=2 --nodekey=93aca5ed922d2e5a69b232a9cfbe1d32dcde3a8d77cd2b482207bbaa5edda834 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=11166 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDXFUNOrCIsMle53J9X19VZOCuyopHwf9oOnfVSxSTctM-KTclgZknbA7AqgoT6SsWPmZUB5ddvnpWEsn83vh6oCgmlkgnY0gmlwhKwSqfiKbXVsdGlhZGRyc5YACASsEqn4BimPAAoErBKp-AYpkN0DgnJzhQACAQAAiXNlY3AyNTZrMaECB6DZs5XtxdIc4gh-xmpx9S3H2T5IqAMzXWqWgRtCGnKDdGNwgimPg3VkcIIpkYV3YWt1MgU
2026-03-01 04:47:01.752 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.89.116 waku 18a6fe088eae39f245b0eefb73d427b1b5ab6f0b6521bc0a89d8fd90247f92e6
2026-03-01 04:47:01.788 DEBUG [src.node.docker_mananger] Container started with ID 18a6fe088eae. Setting up logs at ./log/docker/node5_2026-03-01_04-46-56__0fc668d2-e13b-4dec-91f5-8d2a8c7542ee__wakuorg_nwaku:latest.log
2026-03-01 04:47:01.788 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 11162
2026-03-01 04:47:01.790 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-01 04:47:02.790 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11162/health" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.802 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-01 04:47:02.804 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-01 04:47:02.804 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11162/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.809 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.89.116/tcp/11163/p2p/16Uiu2HAkyeDwgJudcKQQsJH7pitkZopwx9nk6Lxj54gbT7mjsaNx","/ip4/172.18.89.116/tcp/11164/ws/p2p/16Uiu2HAkyeDwgJudcKQQsJH7pitkZopwx9nk6Lxj54gbT7mjsaNx"],"enrUri":"enr:-L24QH3Pnk1eWW1Gc4OP7wTpbRImAyoFR7iQxLq9n_bPsVt1aJtiylGWQCevHLfY7739ognNJpc19ozQOUgOMHnDbTYCgmlkgnY0gmlwhKwSWXSKbXVsdGlhZGRyc5YACASsEll0BiubAAoErBJZdAYrnN0DgnJzhQACAQAAiXNlY3AyNTZrMaECPqXb0607qP3N3H_xubB6GUKA8J88Y8h9Hzc7mOXEKQ2DdGNwgiubg3VkcIIrnYV3YWt1MgE"}'
2026-03-01 04:47:02.809 INFO [src.node.waku_node] REST service is ready !!
2026-03-01 04:47:02.810 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11162/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.169.248/tcp/10639/p2p/16Uiu2HAkuwT6DgBgKzZxNGUUP7LktPsk4VJMWuYa7yCb3UQx7WDf"]'
2026-03-01 04:47:02.815 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.815 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10638/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-01 04:47:02.818 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.818 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22635/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-01 04:47:02.820 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.821 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:57522/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-01 04:47:02.823 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.823 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:17045/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-01 04:47:02.826 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.826 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:11162/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/2/0"]'
2026-03-01 04:47:02.828 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.829 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10638/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d '{"payload": "U2hhcmRpbmcgd29ya3MhIQ==", "contentTopic": "/myapp/1/latest/proto", "timestamp": '$(date +%s%N)'}'
2026-03-01 04:47:02.840 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-01 04:47:02.840 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-01 04:47:02.941 DEBUG [src.steps.sharding] Checking that peer NODE_1:wakuorg/nwaku:latest can find the published message
2026-03-01 04:47:02.942 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10638/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.944 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772340422829188292,"ephemeral":false,"proof":""}]'
2026-03-01 04:47:02.946 DEBUG [src.steps.sharding] Checking that peer NODE_2:wakuorg/nwaku:latest can find the published message
2026-03-01 04:47:02.946 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22635/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.948 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772340422829188292,"ephemeral":false,"proof":""}]'
2026-03-01 04:47:02.949 DEBUG [src.steps.sharding] Checking that peer NODE_3:wakuorg/nwaku:latest can find the published message
2026-03-01 04:47:02.949 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:57522/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.952 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772340422829188292,"ephemeral":false,"proof":""}]'
2026-03-01 04:47:02.953 DEBUG [src.steps.sharding] Checking that peer NODE_4:wakuorg/nwaku:latest can find the published message
2026-03-01 04:47:02.954 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:17045/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.956 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772340422829188292,"ephemeral":false,"proof":""}]'
2026-03-01 04:47:02.957 DEBUG [src.steps.sharding] Checking that peer NODE_5:wakuorg/nwaku:latest can find the published message
2026-03-01 04:47:02.958 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:11162/relay/v1/messages/%2Fwaku%2F2%2Frs%2F2%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-01 04:47:02.960 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"U2hhcmRpbmcgd29ya3MhIQ==","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1772340422829188292,"ephemeral":false,"proof":""}]'
2026-03-01 04:47:02.963 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-01 04:47:02.964 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-01 04:47:02.964 DEBUG [src.node.waku_node] Stopping container with id 838974ce1383
2026-03-01 04:47:03.574 DEBUG [src.node.waku_node] Container stopped.
2026-03-01 04:47:03.575 DEBUG [src.node.waku_node] Stopping container with id bfb4df772653
2026-03-01 04:47:04.153 DEBUG [src.node.waku_node] Container stopped.
2026-03-01 04:47:04.153 DEBUG [src.node.waku_node] Stopping container with id ce6a2b1e1695
2026-03-01 04:47:04.390 ERROR [src.node.docker_mananger] Max retries reached for container 838974ce1383. Exiting log stream.
2026-03-01 04:47:04.709 DEBUG [src.node.waku_node] Container stopped.
2026-03-01 04:47:04.710 DEBUG [src.node.waku_node] Stopping container with id 5369d958cc31
2026-03-01 04:47:04.965 ERROR [src.node.docker_mananger] Max retries reached for container bfb4df772653. Exiting log stream.
2026-03-01 04:47:05.305 DEBUG [src.node.waku_node] Container stopped.
2026-03-01 04:47:05.306 DEBUG [src.node.waku_node] Stopping container with id 18a6fe088eae
2026-03-01 04:47:05.523 ERROR [src.node.docker_mananger] Max retries reached for container ce6a2b1e1695. Exiting log stream.
2026-03-01 04:47:05.872 DEBUG [src.node.waku_node] Container stopped.
2026-03-01 04:47:05.875 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-01 04:47:05.890 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-01 04:47:05.894 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-01 04:47:05.899 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-01 04:47:05.903 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-01 04:47:05.908 DEBUG [src.node.docker_mananger] No errors found in the waku logs.