188 lines
34 KiB
Plaintext

2026-03-22 04:43:17.909 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-22 04:43:17.909 DEBUG [tests.conftest] Running test: test_multiple_receiving_nodes with id: 2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee
2026-03-22 04:43:17.910 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-22 04:43:17.910 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-22 04:43:17.918 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:43:17.918 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node1_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:17.918 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:43:17.918 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:43:17.920 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:43:17.920 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.145.21
2026-03-22 04:43:17.920 DEBUG [src.node.docker_mananger] Generated ports ['7557', '7558', '7559', '7560', '7561']
2026-03-22 04:43:17.920 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:43:17.920 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:43:17.920 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:43:17.921 DEBUG [src.node.docker_mananger] docker run -i -t -p 7557:7557 -p 7558:7558 -p 7559:7559 -p 7560:7560 -p 7561:7561 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=7559 --rest-port=7557 --tcp-port=7558 --discv5-udp-port=7560 --rest-address=0.0.0.0 --nat=extip:172.18.145.21 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=4e25fcabaf1cabeebfeaeaf083d5dbabecdaca6eb35ae7740ca5bbfa8ede84fc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=7561 --metrics-logging=true --lightpush=true --relay=true
2026-03-22 04:43:18.122 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.145.21 waku 4428b208d6340694ff747f54e4978bb8724513028ba49db2a90c36b84f27933b
2026-03-22 04:43:18.148 ERROR [src.node.docker_mananger] Max retries reached for container e4fa9a44788a. Exiting log stream.
2026-03-22 04:43:18.158 DEBUG [src.node.docker_mananger] Container started with ID 4428b208d634. Setting up logs at ./log/docker/receiving_node1_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:18.158 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 7557
2026-03-22 04:43:18.158 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:43:18.681 ERROR [src.node.docker_mananger] Max retries reached for container 272370bd7840. Exiting log stream.
2026-03-22 04:43:19.160 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7557/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:19.163 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:43:19.163 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:43:19.163 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7557/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:19.166 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.145.21/tcp/7558/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R","/ip4/172.18.145.21/tcp/7559/ws/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R"],"enrUri":"enr:-L24QOv6_DS2BLwQiw0WEN_gmWFBw7GmIfE5W3jfCNbmXEQ1Y9qTnYlXkD7TNJNVR7NiePFLJ8uCl-JdGvI-DOCCDfcCgmlkgnY0gmlwhKwSkRWKbXVsdGlhZGRyc5YACASsEpEVBh2GAAoErBKRFQYdh90DgnJzhQADAQAAiXNlY3AyNTZrMaEDp0zHD1B-jMNfsXet_AoQlUp82o_U21jbLjpeztxSpqKDdGNwgh2Gg3VkcIIdiIV3YWt1Mgk"}'
2026-03-22 04:43:19.166 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:43:19.173 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:43:19.173 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node2_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:19.173 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:43:19.173 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:43:19.175 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:43:19.175 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.78.237
2026-03-22 04:43:19.175 DEBUG [src.node.docker_mananger] Generated ports ['24675', '24676', '24677', '24678', '24679']
2026-03-22 04:43:19.175 DEBUG [src.node.waku_node] Cluster id was resolved to: 3
2026-03-22 04:43:19.175 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-22 04:43:19.175 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:43:19.175 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:43:19.176 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:43:19.176 DEBUG [src.node.docker_mananger] docker run -i -t -p 24675:24675 -p 24676:24676 -p 24677:24677 -p 24678:24678 -p 24679:24679 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=24677 --rest-port=24675 --tcp-port=24676 --discv5-udp-port=24678 --rest-address=0.0.0.0 --nat=extip:172.18.78.237 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f975afedb1bfc008aed7ef052c129aefcab7a16d2ee5af44620bd62effea0daa --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=24679 --metrics-logging=true --lightpush=true --relay=true
2026-03-22 04:43:19.369 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.78.237 waku 2d8fa2e60846ac27b73f4ba825efd96ee504962dcd9499c48544ed98ee9d5b62
2026-03-22 04:43:19.406 DEBUG [src.node.docker_mananger] Container started with ID 2d8fa2e60846. Setting up logs at ./log/docker/receiving_node2_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:19.406 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 24675
2026-03-22 04:43:19.406 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:43:20.407 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24675/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:20.410 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:43:20.411 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:43:20.411 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24675/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:20.413 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.78.237/tcp/24676/p2p/16Uiu2HAm5yc3czAjSFv2ZdpCaJ3n2TMJn64ScL4jmdtFsFBaHqBY","/ip4/172.18.78.237/tcp/24677/ws/p2p/16Uiu2HAm5yc3czAjSFv2ZdpCaJ3n2TMJn64ScL4jmdtFsFBaHqBY"],"enrUri":"enr:-L24QN-pQHmvSfN5ss2ChoLOYlHP4f0-AtkKfJQnj-wA5OjPBa3-qiD7NIbb_1hPe_NEEvHJbbGQSIsuBJHutUV0-DkCgmlkgnY0gmlwhKwSTu2KbXVsdGlhZGRyc5YACASsEk7tBmBkAAoErBJO7QZgZd0DgnJzhQADAQAAiXNlY3AyNTZrMaECnMMByDClO3rpyUZOmc9Y8rqcqlCtoajlqC12xRYds3ODdGNwgmBkg3VkcIJgZoV3YWt1Mgk"}'
2026-03-22 04:43:20.414 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:43:20.414 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:24675/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.145.21/tcp/7558/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R"]'
2026-03-22 04:43:20.457 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:20.466 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:43:20.466 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node3_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:20.466 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:43:20.467 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:43:20.468 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:43:20.468 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.79.192
2026-03-22 04:43:20.468 DEBUG [src.node.docker_mananger] Generated ports ['9409', '9410', '9411', '9412', '9413']
2026-03-22 04:43:20.468 DEBUG [src.node.waku_node] Cluster id was resolved to: 3
2026-03-22 04:43:20.469 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-22 04:43:20.469 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:43:20.469 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:43:20.469 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:43:20.469 DEBUG [src.node.docker_mananger] docker run -i -t -p 9409:9409 -p 9410:9410 -p 9411:9411 -p 9412:9412 -p 9413:9413 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=9411 --rest-port=9409 --tcp-port=9410 --discv5-udp-port=9412 --rest-address=0.0.0.0 --nat=extip:172.18.79.192 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=caefec1912f5fb1bc6356045adda5dce7fbe6e83cfdcee0da135ce67f5fd3b37 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=9413 --metrics-logging=true --lightpush=true --relay=true
2026-03-22 04:43:20.674 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.79.192 waku 8adf76c6e8c1ea62dd2437c0867ed9a087661acecc5fab237b3af83f8ec727dc
2026-03-22 04:43:20.715 DEBUG [src.node.docker_mananger] Container started with ID 8adf76c6e8c1. Setting up logs at ./log/docker/receiving_node3_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:20.715 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 9409
2026-03-22 04:43:20.716 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:43:21.716 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9409/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:21.719 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:43:21.719 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:43:21.719 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9409/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:21.722 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.79.192/tcp/9410/p2p/16Uiu2HAm5pMmfQox6t8Q2C8pPaFFXsjBtPYxtDgW8gfNV97ZmHbD","/ip4/172.18.79.192/tcp/9411/ws/p2p/16Uiu2HAm5pMmfQox6t8Q2C8pPaFFXsjBtPYxtDgW8gfNV97ZmHbD"],"enrUri":"enr:-L24QLjA6vzNgxzos_uUR4uTPtgNrTzQt3UouFS3MXthpn3-H3VRVpQ7yRyJUsdB9UzHuX5xmqAzCZJ-M7180_aZtz8CgmlkgnY0gmlwhKwST8CKbXVsdGlhZGRyc5YACASsEk_ABiTCAAoErBJPwAYkw90DgnJzhQADAQAAiXNlY3AyNTZrMaECmmSjg4TY-PtwTh_bTcavHCSJmitZg8c2gC3--dcNqaCDdGNwgiTCg3VkcIIkxIV3YWt1Mgk"}'
2026-03-22 04:43:21.722 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:43:21.723 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:9409/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.145.21/tcp/7558/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R"]'
2026-03-22 04:43:21.767 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:21.767 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:9409/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.78.237/tcp/24676/p2p/16Uiu2HAm5yc3czAjSFv2ZdpCaJ3n2TMJn64ScL4jmdtFsFBaHqBY"]'
2026-03-22 04:43:21.795 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:21.809 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:43:21.809 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/receiving_node4_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:21.810 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:43:21.810 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:43:21.814 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:43:21.815 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.1.93
2026-03-22 04:43:21.815 DEBUG [src.node.docker_mananger] Generated ports ['13297', '13298', '13299', '13300', '13301']
2026-03-22 04:43:21.815 DEBUG [src.node.waku_node] Cluster id was resolved to: 3
2026-03-22 04:43:21.815 DEBUG [src.node.waku_node] Removing pubsub-topic from nwaku args
2026-03-22 04:43:21.816 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:43:21.816 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:43:21.816 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:43:21.817 DEBUG [src.node.docker_mananger] docker run -i -t -p 13297:13297 -p 13298:13298 -p 13299:13299 -p 13300:13300 -p 13301:13301 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=13299 --rest-port=13297 --tcp-port=13298 --discv5-udp-port=13300 --rest-address=0.0.0.0 --nat=extip:172.18.1.93 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=2ddd62dc6c2bca2321bafab66a1a5c6bfb5de18a205bcf1c1d7af1fbf2d4770a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=13301 --metrics-logging=true --lightpush=true --relay=true
2026-03-22 04:43:22.019 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.1.93 waku 716d1c1297592515dcc22254294b1ad50c23b36f403081943655e08cc4e52cdd
2026-03-22 04:43:22.053 DEBUG [src.node.docker_mananger] Container started with ID 716d1c129759. Setting up logs at ./log/docker/receiving_node4_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:22.053 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 13297
2026-03-22 04:43:22.053 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:43:23.055 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13297/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:23.059 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:43:23.059 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:43:23.059 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13297/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:23.061 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.1.93/tcp/13298/p2p/16Uiu2HAmPWBURVhoyn2vRGfs33BkoTvuJAkJTB3A5MZegJo5LXft","/ip4/172.18.1.93/tcp/13299/ws/p2p/16Uiu2HAmPWBURVhoyn2vRGfs33BkoTvuJAkJTB3A5MZegJo5LXft"],"enrUri":"enr:-L24QCQ02bkdnWvwf5hh0c2VgQJT0mKfIl9u3IcZRLI8WfHUBbcpvlLoWFq_jeMpxuBYfZdtRhjovC6t7Z5V5LkxM0sCgmlkgnY0gmlwhKwSAV2KbXVsdGlhZGRyc5YACASsEgFdBjPyAAoErBIBXQYz890DgnJzhQADAQAAiXNlY3AyNTZrMaEDoS7n3J3HpC2DFDS_Lc5rjYGI2_nybyeNFRbubdpqQ7-DdGNwgjPyg3VkcIIz9IV3YWt1Mgk"}'
2026-03-22 04:43:23.061 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:43:23.062 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13297/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.145.21/tcp/7558/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R"]'
2026-03-22 04:43:23.107 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:23.112 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13297/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.78.237/tcp/24676/p2p/16Uiu2HAm5yc3czAjSFv2ZdpCaJ3n2TMJn64ScL4jmdtFsFBaHqBY"]'
2026-03-22 04:43:23.138 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:23.139 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13297/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.79.192/tcp/9410/p2p/16Uiu2HAm5pMmfQox6t8Q2C8pPaFFXsjBtPYxtDgW8gfNV97ZmHbD"]'
2026-03-22 04:43:23.173 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:23.185 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-22 04:43:23.186 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/lightpush_node1_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:23.187 DEBUG [src.node.waku_node] Starting Node...
2026-03-22 04:43:23.188 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-22 04:43:23.190 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-22 04:43:23.191 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.193.149
2026-03-22 04:43:23.191 DEBUG [src.node.docker_mananger] Generated ports ['22057', '22058', '22059', '22060', '22061']
2026-03-22 04:43:23.193 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-22 04:43:23.193 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-22 04:43:23.197 DEBUG [src.node.waku_node] Using volumes []
2026-03-22 04:43:23.197 DEBUG [src.node.docker_mananger] docker run -i -t -p 22057:22057 -p 22058:22058 -p 22059:22059 -p 22060:22060 -p 22061:22061 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=22059 --rest-port=22057 --tcp-port=22058 --discv5-udp-port=22060 --rest-address=0.0.0.0 --nat=extip:172.18.193.149 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=de7bd4f3fecea5fabbb7d7585f80cebf46cffaefcdb5361dce0abebd7abe68ab --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=22061 --metrics-logging=true --discv5-bootstrap-node=enr:-L24QOv6_DS2BLwQiw0WEN_gmWFBw7GmIfE5W3jfCNbmXEQ1Y9qTnYlXkD7TNJNVR7NiePFLJ8uCl-JdGvI-DOCCDfcCgmlkgnY0gmlwhKwSkRWKbXVsdGlhZGRyc5YACASsEpEVBh2GAAoErBKRFQYdh90DgnJzhQADAQAAiXNlY3AyNTZrMaEDp0zHD1B-jMNfsXet_AoQlUp82o_U21jbLjpeztxSpqKDdGNwgh2Gg3VkcIIdiIV3YWt1Mgk --lightpushnode=/ip4/172.18.145.21/tcp/7558/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R --lightpush=true --relay=true
2026-03-22 04:43:23.400 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.193.149 waku 0151689dd77abdfd3f342e9bd649c57a4dc1dd81eeca95fab917973a0caeeb22
2026-03-22 04:43:23.438 DEBUG [src.node.docker_mananger] Container started with ID 0151689dd77a. Setting up logs at ./log/docker/lightpush_node1_2026-03-22_04-43-17__c2624926-cb26-4e78-9f5f-8f0c8de298ee__wakuorg_nwaku:latest.log
2026-03-22 04:43:23.438 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 22057
2026-03-22 04:43:23.439 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-22 04:43:24.442 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22057/health" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.463 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-22 04:43:24.465 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-22 04:43:24.465 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22057/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.470 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.193.149/tcp/22058/p2p/16Uiu2HAmViHayEqmRRJ9EZ71fMURXtwG6q8ZCN8ms5WGS4BagZD3","/ip4/172.18.193.149/tcp/22059/ws/p2p/16Uiu2HAmViHayEqmRRJ9EZ71fMURXtwG6q8ZCN8ms5WGS4BagZD3"],"enrUri":"enr:-L24QPnvNNT9RGxt5wFcV7JdwR6DvpQgbEnRPNTa44HzP17Eeb79PMSCBje-v-JQLUCPbd6HtGAr0rU_SxlnmMIddAACgmlkgnY0gmlwhKwSwZWKbXVsdGlhZGRyc5YACASsEsGVBlYqAAoErBLBlQZWK90DgnJzhQADAQAAiXNlY3AyNTZrMaED_W7oqZcMSJ7T5-Pvw9LzwpitFlmbO9T6SbZdrPEPxOKDdGNwglYqg3VkcIJWLIV3YWt1Mgk"}'
2026-03-22 04:43:24.471 INFO [src.node.waku_node] REST service is ready !!
2026-03-22 04:43:24.471 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22057/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.145.21/tcp/7558/p2p/16Uiu2HAmPv4LbQzczKkpSLRJ3m7YLakFtLLiexY111sYUMSHCz4R"]'
2026-03-22 04:43:24.476 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.477 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22057/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.78.237/tcp/24676/p2p/16Uiu2HAm5yc3czAjSFv2ZdpCaJ3n2TMJn64ScL4jmdtFsFBaHqBY"]'
2026-03-22 04:43:24.500 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.502 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22057/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.79.192/tcp/9410/p2p/16Uiu2HAm5pMmfQox6t8Q2C8pPaFFXsjBtPYxtDgW8gfNV97ZmHbD"]'
2026-03-22 04:43:24.541 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.542 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22057/admin/v1/peers" -H "Content-Type: application/json" -d '["/ip4/172.18.1.93/tcp/13298/p2p/16Uiu2HAmPWBURVhoyn2vRGfs33BkoTvuJAkJTB3A5MZegJo5LXft"]'
2026-03-22 04:43:24.576 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.577 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:7557/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-22 04:43:24.586 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.587 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:24675/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-22 04:43:24.590 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.591 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:9409/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-22 04:43:24.597 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.597 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:13297/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-22 04:43:24.604 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.607 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22057/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/0"]'
2026-03-22 04:43:24.613 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.614 DEBUG [src.steps.light_push] Lightpushing message
2026-03-22 04:43:24.614 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:22057/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/0", "message": {"payload": "TGlnaHQgcHVzaCB3b3JrcyEh", "contentTopic": "/myapp/1/latest/proto", "timestamp": 1774154604614184753}}'
2026-03-22 04:43:24.657 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-22 04:43:24.658 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-22 04:43:24.760 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-03-22 04:43:24.761 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7557/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.763 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774154604614184753,"ephemeral":false,"proof":""}]'
2026-03-22 04:43:24.765 DEBUG [src.steps.light_push] Checking that peer NODE_2:wakuorg/nwaku:latest can find the lightpushed message
2026-03-22 04:43:24.765 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:24675/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.767 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774154604614184753,"ephemeral":false,"proof":""}]'
2026-03-22 04:43:24.769 DEBUG [src.steps.light_push] Checking that peer NODE_3:wakuorg/nwaku:latest can find the lightpushed message
2026-03-22 04:43:24.769 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:9409/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.771 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774154604614184753,"ephemeral":false,"proof":""}]'
2026-03-22 04:43:24.772 DEBUG [src.steps.light_push] Checking that peer NODE_4:wakuorg/nwaku:latest can find the lightpushed message
2026-03-22 04:43:24.773 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:13297/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.775 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774154604614184753,"ephemeral":false,"proof":""}]'
2026-03-22 04:43:24.776 DEBUG [src.steps.light_push] Checking that peer NODE_5:wakuorg/nwaku:latest can find the lightpushed message
2026-03-22 04:43:24.777 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:22057/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F0" -H "Content-Type: application/json" -d 'None'
2026-03-22 04:43:24.779 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"TGlnaHQgcHVzaCB3b3JrcyEh","contentTopic":"/myapp/1/latest/proto","version":0,"timestamp":1774154604614184753,"ephemeral":false,"proof":""}]'
2026-03-22 04:43:24.781 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-22 04:43:24.782 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-22 04:43:24.782 DEBUG [src.node.waku_node] Stopping container with id 4428b208d634
2026-03-22 04:43:25.362 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:43:25.363 DEBUG [src.node.waku_node] Stopping container with id 2d8fa2e60846
2026-03-22 04:43:25.971 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:43:25.973 DEBUG [src.node.waku_node] Stopping container with id 8adf76c6e8c1
2026-03-22 04:43:26.181 ERROR [src.node.docker_mananger] Max retries reached for container 4428b208d634. Exiting log stream.
2026-03-22 04:43:26.561 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:43:26.562 DEBUG [src.node.waku_node] Stopping container with id 716d1c129759
2026-03-22 04:43:26.786 ERROR [src.node.docker_mananger] Max retries reached for container 2d8fa2e60846. Exiting log stream.
2026-03-22 04:43:27.136 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:43:27.138 DEBUG [src.node.waku_node] Stopping container with id 0151689dd77a
2026-03-22 04:43:27.374 ERROR [src.node.docker_mananger] Max retries reached for container 8adf76c6e8c1. Exiting log stream.
2026-03-22 04:43:27.678 DEBUG [src.node.waku_node] Container stopped.
2026-03-22 04:43:27.681 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-22 04:43:27.697 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:43:27.712 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:43:27.726 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:43:27.740 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-22 04:43:27.755 DEBUG [src.node.docker_mananger] No errors found in the waku logs.