179 lines
34 KiB
Plaintext

2025-12-08 12:09:55.119 DEBUG [tests.conftest] Running fixture setup: test_id
2025-12-08 12:09:55.119 DEBUG [tests.conftest] Running test: test_multiple_edge_service_nodes_communication with id: 2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf
2025-12-08 12:09:55.120 DEBUG [src.steps.common] Running fixture setup: common_setup
2025-12-08 12:09:55.120 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2025-12-08 12:09:55.120 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2025-12-08 12:09:55.129 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.129 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.135 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.136 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.142 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.142 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.142 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2025-12-08 12:09:55.142 DEBUG [src.steps.store] Running fixture setup: store_setup
2025-12-08 12:09:55.149 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.149 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.155 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.155 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.160 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.161 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.166 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.166 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.171 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2025-12-08 12:09:55.172 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node8_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:55.172 DEBUG [tests.e2e.test_e2e] Start 2 edges nodes and 3 service nodes
2025-12-08 12:09:55.172 DEBUG [src.node.waku_node] Starting Node...
2025-12-08 12:09:55.172 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-08 12:09:55.207 DEBUG [src.node.docker_mananger] Network waku created
2025-12-08 12:09:55.207 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.232.73
2025-12-08 12:09:55.207 DEBUG [src.node.docker_mananger] Generated ports ['14650', '14651', '14652', '14653', '14654']
2025-12-08 12:09:55.207 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-08 12:09:55.207 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-08 12:09:55.207 DEBUG [src.node.waku_node] Using volumes []
2025-12-08 12:09:55.207 DEBUG [src.node.docker_mananger] docker run -i -t -p 14650:14650 -p 14651:14651 -p 14652:14652 -p 14653:14653 -p 14654:14654 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=14652 --rest-port=14650 --tcp-port=14651 --discv5-udp-port=14653 --rest-address=0.0.0.0 --nat=extip:172.18.232.73 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=784335f64dbffdede6eef3fd017daf895bd14e50d5fb91f704cde655fde0597c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=14654 --metrics-logging=true --relay=true --store=true --lightpush=true
2025-12-08 12:09:57.706 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.232.73 waku c19c621bd4e333f87479aead8f5692337715d59c285965e469c3043717d79c2c
2025-12-08 12:09:57.734 DEBUG [src.node.docker_mananger] Container started with ID c19c621bd4e3. Setting up logs at ./log/docker/node6_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:57.735 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 14650
2025-12-08 12:09:57.735 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-08 12:09:58.736 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14650/health" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:09:58.740 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2025-12-08 12:09:58.740 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-08 12:09:58.740 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14650/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:09:58.742 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.232.73/tcp/14651/p2p/16Uiu2HAmVnJJVdkDXGjSkXYdrpw8suq9nRDSJVTG89K8thHzumSA","/ip4/172.18.232.73/tcp/14652/ws/p2p/16Uiu2HAmVnJJVdkDXGjSkXYdrpw8suq9nRDSJVTG89K8thHzumSA"],"enrUri":"enr:-L24QDBqyPu97r_ocLDPwIqm3qsQtqNmsfh5bZJBqKKC4PmXJXogKxH6n2D8n6ZVtaLEhkTBEaG_6eyvbrYAp18uj04CgmlkgnY0gmlwhKwS6EmKbXVsdGlhZGRyc5YACASsEuhJBjk7AAoErBLoSQY5PN0DgnJzhQADAQAAiXNlY3AyNTZrMaED_nYKpQpStTea1eDOX2wPbdPO4xWCXP35SQfkDSTfzxODdGNwgjk7g3VkcII5PYV3YWt1Mgs"}'
2025-12-08 12:09:58.743 INFO [src.node.waku_node] REST service is ready !!
2025-12-08 12:09:58.743 DEBUG [src.node.waku_node] Starting Node...
2025-12-08 12:09:58.743 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-08 12:09:58.744 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-08 12:09:58.745 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.110.68
2025-12-08 12:09:58.745 DEBUG [src.node.docker_mananger] Generated ports ['29786', '29787', '29788', '29789', '29790']
2025-12-08 12:09:58.745 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-08 12:09:58.745 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-08 12:09:58.745 DEBUG [src.node.waku_node] Using volumes []
2025-12-08 12:09:58.745 DEBUG [src.node.docker_mananger] docker run -i -t -p 29786:29786 -p 29787:29787 -p 29788:29788 -p 29789:29789 -p 29790:29790 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=29788 --rest-port=29786 --tcp-port=29787 --discv5-udp-port=29789 --rest-address=0.0.0.0 --nat=extip:172.18.110.68 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=429b8eba2c2682eefec81af5e6ef3cf1417fd0d5fc815271d5a2ace782c89afd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=29790 --metrics-logging=true --relay=false --lightpushnode=/ip4/172.18.232.73/tcp/14651/p2p/16Uiu2HAmVnJJVdkDXGjSkXYdrpw8suq9nRDSJVTG89K8thHzumSA --discv5-bootstrap-node=enr:-L24QDBqyPu97r_ocLDPwIqm3qsQtqNmsfh5bZJBqKKC4PmXJXogKxH6n2D8n6ZVtaLEhkTBEaG_6eyvbrYAp18uj04CgmlkgnY0gmlwhKwS6EmKbXVsdGlhZGRyc5YACASsEuhJBjk7AAoErBLoSQY5PN0DgnJzhQADAQAAiXNlY3AyNTZrMaED_nYKpQpStTea1eDOX2wPbdPO4xWCXP35SQfkDSTfzxODdGNwgjk7g3VkcII5PYV3YWt1Mgs
2025-12-08 12:09:58.918 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.110.68 waku 215f466ec11f997a883a7f545dbfb8363583972c9bd7366f42ecdd3f0d43648f
2025-12-08 12:09:58.943 DEBUG [src.node.docker_mananger] Container started with ID 215f466ec11f. Setting up logs at ./log/docker/node4_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:09:58.944 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 29786
2025-12-08 12:09:58.944 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-08 12:09:59.945 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29786/health" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:09:59.949 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2025-12-08 12:09:59.949 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-08 12:09:59.949 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:29786/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:09:59.951 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.110.68/tcp/29787/p2p/16Uiu2HAmPL9wemaXs8xGNQ8CvzdphgikjZsANQEYkXiWCkVFkQcB","/ip4/172.18.110.68/tcp/29788/ws/p2p/16Uiu2HAmPL9wemaXs8xGNQ8CvzdphgikjZsANQEYkXiWCkVFkQcB"],"enrUri":"enr:-L24QB_FKGOswDvgbYwx7RFsDIukmBYk0L6MRQpx-7owlHuKBnQIybUl983oj8r2f2qGZM-H2QUj4iLjyMyMqD_PEbYCgmlkgnY0gmlwhKwSbkSKbXVsdGlhZGRyc5YACASsEm5EBnRbAAoErBJuRAZ0XN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDnp1d9g0PNBXvhxHEAeGk2nb_JV8AkllC_jEcstfuVoyDdGNwgnRbg3VkcIJ0XYV3YWt1MgA"}'
2025-12-08 12:09:59.952 INFO [src.node.waku_node] REST service is ready !!
2025-12-08 12:09:59.952 DEBUG [src.node.waku_node] Starting Node...
2025-12-08 12:09:59.952 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-08 12:09:59.953 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-08 12:09:59.953 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.185.140
2025-12-08 12:09:59.953 DEBUG [src.node.docker_mananger] Generated ports ['16021', '16022', '16023', '16024', '16025']
2025-12-08 12:09:59.953 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-08 12:09:59.954 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-08 12:09:59.954 DEBUG [src.node.waku_node] Using volumes []
2025-12-08 12:09:59.954 DEBUG [src.node.docker_mananger] docker run -i -t -p 16021:16021 -p 16022:16022 -p 16023:16023 -p 16024:16024 -p 16025:16025 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=16023 --rest-port=16021 --tcp-port=16022 --discv5-udp-port=16024 --rest-address=0.0.0.0 --nat=extip:172.18.185.140 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=8ddead34626bc2340e5ab66df04d29bba8a8d0b100685b932a55ebf273bc52b2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=16025 --metrics-logging=true --relay=true --store=true --discv5-bootstrap-node=enr:-L24QDBqyPu97r_ocLDPwIqm3qsQtqNmsfh5bZJBqKKC4PmXJXogKxH6n2D8n6ZVtaLEhkTBEaG_6eyvbrYAp18uj04CgmlkgnY0gmlwhKwS6EmKbXVsdGlhZGRyc5YACASsEuhJBjk7AAoErBLoSQY5PN0DgnJzhQADAQAAiXNlY3AyNTZrMaED_nYKpQpStTea1eDOX2wPbdPO4xWCXP35SQfkDSTfzxODdGNwgjk7g3VkcII5PYV3YWt1Mgs
2025-12-08 12:10:00.117 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.185.140 waku 62632785acd726ffe60c7c94051f949b73d1b125b033ca245e87e28b4def35bc
2025-12-08 12:10:00.141 DEBUG [src.node.docker_mananger] Container started with ID 62632785acd7. Setting up logs at ./log/docker/node7_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:10:00.141 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 16021
2025-12-08 12:10:00.141 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-08 12:10:01.142 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16021/health" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:01.146 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2025-12-08 12:10:01.146 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-08 12:10:01.146 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16021/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:01.148 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.185.140/tcp/16022/p2p/16Uiu2HAm8yFUerpD55M8RSPFvUufrdaxF4bWjCb6pEmS39JNc4j7","/ip4/172.18.185.140/tcp/16023/ws/p2p/16Uiu2HAm8yFUerpD55M8RSPFvUufrdaxF4bWjCb6pEmS39JNc4j7"],"enrUri":"enr:-L24QOPbVC1VYBjR66tdD7H-tlRvW7v7Z7k5nGENmyHCHgzTIs5W5Cf7YqSVCOSFhCIBZyww1m6qtvVLStzCoGaSjHQCgmlkgnY0gmlwhKwSuYyKbXVsdGlhZGRyc5YACASsErmMBj6WAAoErBK5jAY-l90DgnJzhQADAQAAiXNlY3AyNTZrMaECyT7Pm3nq31pYB8tTiMYaFeZitfHNhOF7M-s7lVgrWl6DdGNwgj6Wg3VkcII-mIV3YWt1MgM"}'
2025-12-08 12:10:01.149 INFO [src.node.waku_node] REST service is ready !!
2025-12-08 12:10:01.149 DEBUG [src.node.waku_node] Starting Node...
2025-12-08 12:10:01.149 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-08 12:10:01.150 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-08 12:10:01.150 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.78.210
2025-12-08 12:10:01.151 DEBUG [src.node.docker_mananger] Generated ports ['4825', '4826', '4827', '4828', '4829']
2025-12-08 12:10:01.151 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-08 12:10:01.151 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-08 12:10:01.151 DEBUG [src.node.waku_node] Using volumes []
2025-12-08 12:10:01.151 DEBUG [src.node.docker_mananger] docker run -i -t -p 4825:4825 -p 4826:4826 -p 4827:4827 -p 4828:4828 -p 4829:4829 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4827 --rest-port=4825 --tcp-port=4826 --discv5-udp-port=4828 --rest-address=0.0.0.0 --nat=extip:172.18.78.210 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=239f1259cbeff4ffbfc8e9c4b5cad39aed0366f34dd5e4b2b8ecaafeddbedb1c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4829 --metrics-logging=true --relay=true --filter=true --storenode=/ip4/172.18.185.140/tcp/16022/p2p/16Uiu2HAm8yFUerpD55M8RSPFvUufrdaxF4bWjCb6pEmS39JNc4j7 --discv5-bootstrap-node=enr:-L24QOPbVC1VYBjR66tdD7H-tlRvW7v7Z7k5nGENmyHCHgzTIs5W5Cf7YqSVCOSFhCIBZyww1m6qtvVLStzCoGaSjHQCgmlkgnY0gmlwhKwSuYyKbXVsdGlhZGRyc5YACASsErmMBj6WAAoErBK5jAY-l90DgnJzhQADAQAAiXNlY3AyNTZrMaECyT7Pm3nq31pYB8tTiMYaFeZitfHNhOF7M-s7lVgrWl6DdGNwgj6Wg3VkcII-mIV3YWt1MgM
2025-12-08 12:10:01.330 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.78.210 waku 2a2c06b058953fd467a43d61a8a9b749cbb4c3d512fcbeb0a0bbb0c5d9c59049
2025-12-08 12:10:01.355 DEBUG [src.node.docker_mananger] Container started with ID 2a2c06b05895. Setting up logs at ./log/docker/node8_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:10:01.356 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 4825
2025-12-08 12:10:01.356 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-08 12:10:02.356 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4825/health" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:02.361 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"READY"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2025-12-08 12:10:02.361 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-08 12:10:02.361 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4825/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:02.364 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.78.210/tcp/4826/p2p/16Uiu2HAmS8n8MZGNVLeb5wdhiogNnpNeRvXTG9psM4FPg64cGBPA","/ip4/172.18.78.210/tcp/4827/ws/p2p/16Uiu2HAmS8n8MZGNVLeb5wdhiogNnpNeRvXTG9psM4FPg64cGBPA"],"enrUri":"enr:-L24QGPo3rNymIjUDoiV4UWBW5n7BKKG5GLSzpMq6YrovLlrbiOZsjMfWdoMiCrTmlRYsixsGTn6uEpyaVSIh20NS7MCgmlkgnY0gmlwhKwSTtKKbXVsdGlhZGRyc5YACASsEk7SBhLaAAoErBJO0gYS290DgnJzhQADAQAAiXNlY3AyNTZrMaEDyEZfQY3mAUHd_i9E2r8e_D7L22pZcmFZpGong5LYwZWDdGNwghLag3VkcIIS3IV3YWt1MgU"}'
2025-12-08 12:10:02.364 INFO [src.node.waku_node] REST service is ready !!
2025-12-08 12:10:02.364 DEBUG [src.node.waku_node] Starting Node...
2025-12-08 12:10:02.364 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2025-12-08 12:10:02.366 DEBUG [src.node.docker_mananger] Network waku already exists
2025-12-08 12:10:02.366 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.207.222
2025-12-08 12:10:02.366 DEBUG [src.node.docker_mananger] Generated ports ['47898', '47899', '47900', '47901', '47902']
2025-12-08 12:10:02.366 DEBUG [src.node.waku_node] RLN credentials were not set
2025-12-08 12:10:02.367 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2025-12-08 12:10:02.367 DEBUG [src.node.waku_node] Using volumes []
2025-12-08 12:10:02.367 DEBUG [src.node.docker_mananger] docker run -i -t -p 47898:47898 -p 47899:47899 -p 47900:47900 -p 47901:47901 -p 47902:47902 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=47900 --rest-port=47898 --tcp-port=47899 --discv5-udp-port=47901 --rest-address=0.0.0.0 --nat=extip:172.18.207.222 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=483ec2bf41a8e7ead2cad0df5a2cc34060eaafee2c4bcf4a820d85c86bd287d3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=47902 --metrics-logging=true --relay=false --filternode=/ip4/172.18.78.210/tcp/4826/p2p/16Uiu2HAmS8n8MZGNVLeb5wdhiogNnpNeRvXTG9psM4FPg64cGBPA --storenode=/ip4/172.18.185.140/tcp/16022/p2p/16Uiu2HAm8yFUerpD55M8RSPFvUufrdaxF4bWjCb6pEmS39JNc4j7 --discv5-bootstrap-node=enr:-L24QOPbVC1VYBjR66tdD7H-tlRvW7v7Z7k5nGENmyHCHgzTIs5W5Cf7YqSVCOSFhCIBZyww1m6qtvVLStzCoGaSjHQCgmlkgnY0gmlwhKwSuYyKbXVsdGlhZGRyc5YACASsErmMBj6WAAoErBK5jAY-l90DgnJzhQADAQAAiXNlY3AyNTZrMaECyT7Pm3nq31pYB8tTiMYaFeZitfHNhOF7M-s7lVgrWl6DdGNwgj6Wg3VkcII-mIV3YWt1MgM
2025-12-08 12:10:02.551 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.207.222 waku 23e2b5ad5625a0088d827193cbc254dc314edb529985ee88d29b8c30f84960e3
2025-12-08 12:10:02.577 DEBUG [src.node.docker_mananger] Container started with ID 23e2b5ad5625. Setting up logs at ./log/docker/node5_2025-12-08_12-09-55__fc773d4a-c6d3-4f8d-853b-7726871044bf__wakuorg_nwaku:latest.log
2025-12-08 12:10:02.577 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 47898
2025-12-08 12:10:02.577 DEBUG [src.libs.common] Sleeping for 1 seconds
2025-12-08 12:10:03.578 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47898/health" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:03.582 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"READY"},{"Filter Client":"READY"}]}'
2025-12-08 12:10:03.582 INFO [src.node.waku_node] Node protocols are initialized !!
2025-12-08 12:10:03.582 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47898/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:03.585 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.207.222/tcp/47899/p2p/16Uiu2HAm9P88NyaYBYaDejDQnKMRbTsEWCoWdYMrWnuqi3Re2Z1V","/ip4/172.18.207.222/tcp/47900/ws/p2p/16Uiu2HAm9P88NyaYBYaDejDQnKMRbTsEWCoWdYMrWnuqi3Re2Z1V"],"enrUri":"enr:-L24QDhQJjDqo0NjveMeLAVdZMdeZ7E_5Swm2ybunnUzrnzXcNXldilOGFDoHp10uLHLzjSSCH6oT696INC8eJ3iiboCgmlkgnY0gmlwhKwSz96KbXVsdGlhZGRyc5YACASsEs_eBrsbAAoErBLP3ga7HN0DgnJzhQADAQAAiXNlY3AyNTZrMaECz1xwsmb3caZVxaRN9UjdJzjLbuYnmHXmOSsQ8w476BSDdGNwgrsbg3VkcIK7HYV3YWt1MgA"}'
2025-12-08 12:10:03.585 INFO [src.node.waku_node] REST service is ready !!
2025-12-08 12:10:03.585 DEBUG [tests.e2e.test_e2e] Connect 3 service nodes to relay subscriptions
2025-12-08 12:10:03.585 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:14650/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2025-12-08 12:10:03.588 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-08 12:10:03.589 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:16021/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2025-12-08 12:10:03.597 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-08 12:10:03.597 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4825/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2025-12-08 12:10:03.601 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-08 12:10:03.602 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14650/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:03.606 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.185.140/tcp/34210/p2p/16Uiu2HAm8yFUerpD55M8RSPFvUufrdaxF4bWjCb6pEmS39JNc4j7","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-109-ga8590a","origin":"Discv5"}]'
2025-12-08 12:10:03.606 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:16021/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:03.609 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.232.73/tcp/14651/p2p/16Uiu2HAmVnJJVdkDXGjSkXYdrpw8suq9nRDSJVTG89K8thHzumSA","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/lightpush/3.0.0","/vac/waku/lightpush/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-109-ga8590a","origin":"Discv5"},{"multiaddr":"/ip4/172.17.0.5/tcp/54616/p2p/16Uiu2HAmS8n8MZGNVLeb5wdhiogNnpNeRvXTG9psM4FPg64cGBPA","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-subscribe/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-109-ga8590a","origin":"UnknownOrigin"}]'
2025-12-08 12:10:03.609 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4825/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:03.611 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.185.140/tcp/16022/p2p/16Uiu2HAm8yFUerpD55M8RSPFvUufrdaxF4bWjCb6pEmS39JNc4j7","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-109-ga8590a","origin":"Discv5"}]'
2025-12-08 12:10:03.611 DEBUG [src.libs.common] Sleeping for 30 seconds
2025-12-08 12:10:33.612 DEBUG [tests.e2e.test_e2e] Edge node2 makes filter subscription to pubsubtopic /waku/2/rs/3/1 and content topic /test/1/waku-filter/proto
2025-12-08 12:10:33.612 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:47898/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "1", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2025-12-08 12:10:33.650 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1","statusDesc":"OK"}'
2025-12-08 12:10:33.653 DEBUG [tests.e2e.test_e2e] Check if service node1 receives message sent by edge node1
2025-12-08 12:10:33.653 DEBUG [src.steps.light_push] Lightpushing message
2025-12-08 12:10:33.653 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:29786/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/1", "message": {"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": 1765195833653342372}}'
2025-12-08 12:10:33.692 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2025-12-08 12:10:33.694 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2025-12-08 12:10:33.795 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2025-12-08 12:10:33.796 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:14650/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:33.799 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1765195833653342372,"ephemeral":false,"proof":""}]'
2025-12-08 12:10:33.801 DEBUG [tests.e2e.test_e2e] Check if edge node2 can query stored message
2025-12-08 12:10:33.801 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2025-12-08 12:10:33.801 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47898/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:33.827 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"7f0ec718dbe60d152228","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xa768969cc8db00a49747279fd858b1125752cc39faeff8f76b05da9c254fac4c"}]}'
2025-12-08 12:10:33.828 DEBUG [src.steps.store] messages length is 1
2025-12-08 12:10:33.828 DEBUG [tests.e2e.test_e2e] Check if service node3 can query stored message
2025-12-08 12:10:33.828 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2025-12-08 12:10:33.829 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4825/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:33.838 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"ab10561ff8a4c9fa8496","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xa768969cc8db00a49747279fd858b1125752cc39faeff8f76b05da9c254fac4c"}]}'
2025-12-08 12:10:33.839 DEBUG [src.steps.store] messages length is 1
2025-12-08 12:10:33.839 DEBUG [tests.e2e.test_e2e] Check if edge node2 can get sent message using filter get request
2025-12-08 12:10:33.840 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:47898/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2025-12-08 12:10:33.842 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1765195833653342372,"ephemeral":false}]'
2025-12-08 12:10:33.844 DEBUG [tests.conftest] Running fixture teardown: test_setup
2025-12-08 12:10:33.845 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2025-12-08 12:10:33.845 DEBUG [src.node.waku_node] Stopping container with id c19c621bd4e3
2025-12-08 12:10:34.362 DEBUG [src.node.waku_node] Container stopped.
2025-12-08 12:10:34.364 DEBUG [src.node.waku_node] Stopping container with id 215f466ec11f
2025-12-08 12:10:34.841 DEBUG [src.node.waku_node] Container stopped.
2025-12-08 12:10:34.841 DEBUG [src.node.waku_node] Stopping container with id 62632785acd7
2025-12-08 12:10:35.250 ERROR [src.node.docker_mananger] Max retries reached for container c19c621bd4e3. Exiting log stream.
2025-12-08 12:10:35.336 DEBUG [src.node.waku_node] Container stopped.
2025-12-08 12:10:35.336 DEBUG [src.node.waku_node] Stopping container with id 2a2c06b05895
2025-12-08 12:10:35.653 ERROR [src.node.docker_mananger] Max retries reached for container 215f466ec11f. Exiting log stream.
2025-12-08 12:10:35.793 DEBUG [src.node.waku_node] Container stopped.
2025-12-08 12:10:35.794 DEBUG [src.node.waku_node] Stopping container with id 23e2b5ad5625
2025-12-08 12:10:36.149 ERROR [src.node.docker_mananger] Max retries reached for container 62632785acd7. Exiting log stream.
2025-12-08 12:10:36.279 DEBUG [src.node.waku_node] Container stopped.
2025-12-08 12:10:36.280 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2025-12-08 12:10:36.327 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-08 12:10:36.333 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-08 12:10:36.359 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-08 12:10:36.384 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2025-12-08 12:10:36.392 DEBUG [src.node.docker_mananger] No errors found in the waku logs.