179 lines
34 KiB
Plaintext

2026-03-13 04:32:27.496 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:32:27.497 DEBUG [tests.conftest] Running test: test_multiple_edge_service_nodes_communication with id: 2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294
2026-03-13 04:32:27.497 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:32:27.497 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-03-13 04:32:27.497 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-03-13 04:32:27.507 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.507 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.514 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.514 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.521 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.521 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.521 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:32:27.521 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-03-13 04:32:27.528 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.529 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.535 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.535 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.541 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.541 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.547 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.547 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.554 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:32:27.554 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node8_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:27.554 DEBUG [tests.e2e.test_e2e] Start 2 edges nodes and 3 service nodes
2026-03-13 04:32:27.554 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:32:27.554 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:32:27.601 DEBUG [src.node.docker_mananger] Network waku created
2026-03-13 04:32:27.601 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.136.92
2026-03-13 04:32:27.601 DEBUG [src.node.docker_mananger] Generated ports ['65309', '65310', '65311', '65312', '65313']
2026-03-13 04:32:27.601 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:32:27.601 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:32:27.601 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:32:27.602 DEBUG [src.node.docker_mananger] docker run -i -t -p 65309:65309 -p 65310:65310 -p 65311:65311 -p 65312:65312 -p 65313:65313 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=65311 --rest-port=65309 --tcp-port=65310 --discv5-udp-port=65312 --rest-address=0.0.0.0 --nat=extip:172.18.136.92 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=c9c6ed9f73360db63a3faa3f4a2eda0a1d707e8a1b00c4e935516c2c7bf73ff0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=65313 --metrics-logging=true --relay=true --store=true --lightpush=true
2026-03-13 04:32:30.613 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.136.92 waku 23c6504139a718dae8ba237cb58ae061bfeb81992864d4d1df9e0fa26306b538
2026-03-13 04:32:30.654 DEBUG [src.node.docker_mananger] Container started with ID 23c6504139a7. Setting up logs at ./log/docker/node6_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:30.654 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 65309
2026-03-13 04:32:30.655 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:32:31.655 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65309/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:31.659 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:32:31.659 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:32:31.659 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65309/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:31.662 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.136.92/tcp/65310/p2p/16Uiu2HAmUyZJQmaJoizZJWBxLCbQ5y5ELXxX2FzpGtQC5DdxcxWc","/ip4/172.18.136.92/tcp/65311/ws/p2p/16Uiu2HAmUyZJQmaJoizZJWBxLCbQ5y5ELXxX2FzpGtQC5DdxcxWc"],"enrUri":"enr:-L24QDJyLJbh0KjhjaojMWO3scgtD9-Yrk3WlR0syX5pOTXgD80IWyHrQufToQZ_9DmExD9oV0zQYAjMtQ80HVjY3N0CgmlkgnY0gmlwhKwSiFyKbXVsdGlhZGRyc5YACASsEohcBv8eAAoErBKIXAb_H90DgnJzhQADAQAAiXNlY3AyNTZrMaED8nyz1UeNWymVVBILGKWrKPfzeg7pVLuKVtUnolqXr9mDdGNwgv8eg3VkcIL_IIV3YWt1Mgs"}'
2026-03-13 04:32:31.662 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:32:31.662 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:32:31.662 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:32:31.664 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:32:31.664 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.212.26
2026-03-13 04:32:31.664 DEBUG [src.node.docker_mananger] Generated ports ['40892', '40893', '40894', '40895', '40896']
2026-03-13 04:32:31.664 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:32:31.664 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:32:31.664 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:32:31.665 DEBUG [src.node.docker_mananger] docker run -i -t -p 40892:40892 -p 40893:40893 -p 40894:40894 -p 40895:40895 -p 40896:40896 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=40894 --rest-port=40892 --tcp-port=40893 --discv5-udp-port=40895 --rest-address=0.0.0.0 --nat=extip:172.18.212.26 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=9d5184ecc1f48fc1eb5aadc5971a0e1bff1aee2afcedfdeefb37c4d368879eef --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=40896 --metrics-logging=true --relay=false --lightpushnode=/ip4/172.18.136.92/tcp/65310/p2p/16Uiu2HAmUyZJQmaJoizZJWBxLCbQ5y5ELXxX2FzpGtQC5DdxcxWc --discv5-bootstrap-node=enr:-L24QDJyLJbh0KjhjaojMWO3scgtD9-Yrk3WlR0syX5pOTXgD80IWyHrQufToQZ_9DmExD9oV0zQYAjMtQ80HVjY3N0CgmlkgnY0gmlwhKwSiFyKbXVsdGlhZGRyc5YACASsEohcBv8eAAoErBKIXAb_H90DgnJzhQADAQAAiXNlY3AyNTZrMaED8nyz1UeNWymVVBILGKWrKPfzeg7pVLuKVtUnolqXr9mDdGNwgv8eg3VkcIL_IIV3YWt1Mgs
2026-03-13 04:32:31.868 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.212.26 waku 337da2ea84848ff6c9933731ec9fec1b68e301c1a797968c61891e2cf7aea033
2026-03-13 04:32:31.901 DEBUG [src.node.docker_mananger] Container started with ID 337da2ea8484. Setting up logs at ./log/docker/node4_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:31.901 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 40892
2026-03-13 04:32:31.901 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:32:32.902 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40892/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:32.905 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:32:32.905 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:32:32.905 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:40892/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:32.908 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.212.26/tcp/40893/p2p/16Uiu2HAm2YxxJA1wU1UjtenanCFv921NuaJZhNG9dXUGDWfJay9C","/ip4/172.18.212.26/tcp/40894/ws/p2p/16Uiu2HAm2YxxJA1wU1UjtenanCFv921NuaJZhNG9dXUGDWfJay9C"],"enrUri":"enr:-L24QDZiI6XUBqr-BrtlK964fbLGopYA45F9lUsGT2QYmqNFOGza3KRByXMk0QHhwh1uJIxz9ineJZalUfioqVEa-gICgmlkgnY0gmlwhKwS1BqKbXVsdGlhZGRyc5YACASsEtQaBp-9AAoErBLUGgafvt0DgnJzhQADAQAAiXNlY3AyNTZrMaECaeARIiFslW6OXuKHN2XLgug31rFHkcZYn8k6ASlSLPODdGNwgp-9g3VkcIKfv4V3YWt1MgA"}'
2026-03-13 04:32:32.908 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:32:32.908 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:32:32.908 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:32:32.910 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:32:32.910 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.158.229
2026-03-13 04:32:32.910 DEBUG [src.node.docker_mananger] Generated ports ['33594', '33595', '33596', '33597', '33598']
2026-03-13 04:32:32.910 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:32:32.910 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:32:32.910 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:32:32.911 DEBUG [src.node.docker_mananger] docker run -i -t -p 33594:33594 -p 33595:33595 -p 33596:33596 -p 33597:33597 -p 33598:33598 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33596 --rest-port=33594 --tcp-port=33595 --discv5-udp-port=33597 --rest-address=0.0.0.0 --nat=extip:172.18.158.229 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f8a6f099d92a7cad133bdfb5ee2eeff3ddaa3fcf3a9e62bcf6ac3100edffca3d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33598 --metrics-logging=true --relay=true --store=true --discv5-bootstrap-node=enr:-L24QDJyLJbh0KjhjaojMWO3scgtD9-Yrk3WlR0syX5pOTXgD80IWyHrQufToQZ_9DmExD9oV0zQYAjMtQ80HVjY3N0CgmlkgnY0gmlwhKwSiFyKbXVsdGlhZGRyc5YACASsEohcBv8eAAoErBKIXAb_H90DgnJzhQADAQAAiXNlY3AyNTZrMaED8nyz1UeNWymVVBILGKWrKPfzeg7pVLuKVtUnolqXr9mDdGNwgv8eg3VkcIL_IIV3YWt1Mgs
2026-03-13 04:32:33.111 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.158.229 waku eff452bcb466e65a8eb7f911601657a14bdcdad752fc0105b0e585a928c8b7f5
2026-03-13 04:32:33.144 DEBUG [src.node.docker_mananger] Container started with ID eff452bcb466. Setting up logs at ./log/docker/node7_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:33.145 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33594
2026-03-13 04:32:33.145 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:32:34.146 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33594/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:34.150 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:32:34.150 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:32:34.150 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33594/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:34.153 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.158.229/tcp/33595/p2p/16Uiu2HAkz32XHRtUtc6xziZKLmUtKQx5Sn4fC8eFP3LKxPPNXTCw","/ip4/172.18.158.229/tcp/33596/ws/p2p/16Uiu2HAkz32XHRtUtc6xziZKLmUtKQx5Sn4fC8eFP3LKxPPNXTCw"],"enrUri":"enr:-L24QIghf7EOB5miH5lFopEgje_lrdYYP0x-xeqAwSUKz7LPNnyrLDNmFmXZfVX1-3kjpI6b9Btr4qA__f90IqVzcdECgmlkgnY0gmlwhKwSnuWKbXVsdGlhZGRyc5YACASsEp7lBoM7AAoErBKe5QaDPN0DgnJzhQADAQAAiXNlY3AyNTZrMaECRH1N0vOpXmOwv-FvbAaXXMNby-JTh2AEgn9AS9iWuhyDdGNwgoM7g3VkcIKDPYV3YWt1MgM"}'
2026-03-13 04:32:34.153 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:32:34.153 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:32:34.153 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:32:34.155 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:32:34.155 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.107.188
2026-03-13 04:32:34.155 DEBUG [src.node.docker_mananger] Generated ports ['21843', '21844', '21845', '21846', '21847']
2026-03-13 04:32:34.155 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:32:34.156 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:32:34.156 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:32:34.156 DEBUG [src.node.docker_mananger] docker run -i -t -p 21843:21843 -p 21844:21844 -p 21845:21845 -p 21846:21846 -p 21847:21847 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=21845 --rest-port=21843 --tcp-port=21844 --discv5-udp-port=21846 --rest-address=0.0.0.0 --nat=extip:172.18.107.188 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e33d099bcf6588005b7a64df9d5eed2f3ab2dbbb2cfad837b6a2e37af3a276da --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=21847 --metrics-logging=true --relay=true --filter=true --storenode=/ip4/172.18.158.229/tcp/33595/p2p/16Uiu2HAkz32XHRtUtc6xziZKLmUtKQx5Sn4fC8eFP3LKxPPNXTCw --discv5-bootstrap-node=enr:-L24QIghf7EOB5miH5lFopEgje_lrdYYP0x-xeqAwSUKz7LPNnyrLDNmFmXZfVX1-3kjpI6b9Btr4qA__f90IqVzcdECgmlkgnY0gmlwhKwSnuWKbXVsdGlhZGRyc5YACASsEp7lBoM7AAoErBKe5QaDPN0DgnJzhQADAQAAiXNlY3AyNTZrMaECRH1N0vOpXmOwv-FvbAaXXMNby-JTh2AEgn9AS9iWuhyDdGNwgoM7g3VkcIKDPYV3YWt1MgM
2026-03-13 04:32:34.392 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.107.188 waku d27bc20e7d6f438b3be7afd4d8bc581f8213ef524bc897010e12757a2d73a2a0
2026-03-13 04:32:34.428 DEBUG [src.node.docker_mananger] Container started with ID d27bc20e7d6f. Setting up logs at ./log/docker/node8_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:34.429 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 21843
2026-03-13 04:32:34.429 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:32:35.430 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21843/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:35.433 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:32:35.434 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:32:35.434 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21843/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:35.436 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.107.188/tcp/21844/p2p/16Uiu2HAm6sKWQgwCqahcDxwnTnpt3PMqp4hgmsBLJHwVtr6x26iJ","/ip4/172.18.107.188/tcp/21845/ws/p2p/16Uiu2HAm6sKWQgwCqahcDxwnTnpt3PMqp4hgmsBLJHwVtr6x26iJ"],"enrUri":"enr:-L24QOua0gFupkt6sCDFtZ3sOCoj3OZPAWHBzVVoNN8B0Ui_Y4MSZqr34d9rSiFEFVVvxDmEHmDkiasIgU2a6sM39LUCgmlkgnY0gmlwhKwSa7yKbXVsdGlhZGRyc5YACASsEmu8BlVUAAoErBJrvAZVVd0DgnJzhQADAQAAiXNlY3AyNTZrMaECqgKCYYBr5e2DsLa17prHRG2_kNOq62DgPBk8jI_S-YeDdGNwglVUg3VkcIJVVoV3YWt1MgU"}'
2026-03-13 04:32:35.436 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:32:35.437 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:32:35.437 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:32:35.438 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:32:35.438 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.127.161
2026-03-13 04:32:35.438 DEBUG [src.node.docker_mananger] Generated ports ['56494', '56495', '56496', '56497', '56498']
2026-03-13 04:32:35.439 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:32:35.439 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:32:35.439 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:32:35.439 DEBUG [src.node.docker_mananger] docker run -i -t -p 56494:56494 -p 56495:56495 -p 56496:56496 -p 56497:56497 -p 56498:56498 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=56496 --rest-port=56494 --tcp-port=56495 --discv5-udp-port=56497 --rest-address=0.0.0.0 --nat=extip:172.18.127.161 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ba30407b0f2c41a8d7acfb9ca4a84f5eb6db0bdaab4ed30be0a463fb56ecb8d8 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=56498 --metrics-logging=true --relay=false --filternode=/ip4/172.18.107.188/tcp/21844/p2p/16Uiu2HAm6sKWQgwCqahcDxwnTnpt3PMqp4hgmsBLJHwVtr6x26iJ --storenode=/ip4/172.18.158.229/tcp/33595/p2p/16Uiu2HAkz32XHRtUtc6xziZKLmUtKQx5Sn4fC8eFP3LKxPPNXTCw --discv5-bootstrap-node=enr:-L24QIghf7EOB5miH5lFopEgje_lrdYYP0x-xeqAwSUKz7LPNnyrLDNmFmXZfVX1-3kjpI6b9Btr4qA__f90IqVzcdECgmlkgnY0gmlwhKwSnuWKbXVsdGlhZGRyc5YACASsEp7lBoM7AAoErBKe5QaDPN0DgnJzhQADAQAAiXNlY3AyNTZrMaECRH1N0vOpXmOwv-FvbAaXXMNby-JTh2AEgn9AS9iWuhyDdGNwgoM7g3VkcIKDPYV3YWt1MgM
2026-03-13 04:32:35.662 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.127.161 waku d75fdd256e1ea980a7686731864ab33556b8c7902047cf058c88c3e0e14ac7e9
2026-03-13 04:32:35.696 DEBUG [src.node.docker_mananger] Container started with ID d75fdd256e1e. Setting up logs at ./log/docker/node5_2026-03-13_04-32-27__6b90288d-cc81-41b2-ab03-abec2790b294__wakuorg_nwaku:latest.log
2026-03-13 04:32:35.697 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 56494
2026-03-13 04:32:35.697 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:32:36.697 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56494/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:36.701 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:32:36.701 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:32:36.701 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56494/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:36.704 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.127.161/tcp/56495/p2p/16Uiu2HAmHe7ppVLntxoP3CSepAuA2QmKWwpHHox8Ru7qTSDoK5cG","/ip4/172.18.127.161/tcp/56496/ws/p2p/16Uiu2HAmHe7ppVLntxoP3CSepAuA2QmKWwpHHox8Ru7qTSDoK5cG"],"enrUri":"enr:-L24QK7OXWGXlvb_DHz7ttO-9no4j98H7zar27duZ8_ve_2oAASm-YCY-1hH2WlQ8yrX_PZpBed_Nf42pstUWMeuocMCgmlkgnY0gmlwhKwSf6GKbXVsdGlhZGRyc5YACASsEn-hBtyvAAoErBJ_oQbcsN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDShFOMReCJ441Dc-TWQuJCzFeUZiuOfaTIFXFXSE86P2DdGNwgtyvg3VkcILcsYV3YWt1MgA"}'
2026-03-13 04:32:36.704 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:32:36.704 DEBUG [tests.e2e.test_e2e] Connect 3 service nodes to relay subscriptions
2026-03-13 04:32:36.704 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:65309/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:32:36.708 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:32:36.709 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33594/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:32:36.716 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:32:36.717 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:21843/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:32:36.722 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:32:36.723 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65309/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:36.727 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.158.229/tcp/57230/p2p/16Uiu2HAkz32XHRtUtc6xziZKLmUtKQx5Sn4fC8eFP3LKxPPNXTCw","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:32:36.728 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33594/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:36.730 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.136.92/tcp/65310/p2p/16Uiu2HAmUyZJQmaJoizZJWBxLCbQ5y5ELXxX2FzpGtQC5DdxcxWc","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/lightpush/3.0.0","/vac/waku/lightpush/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.17.0.5/tcp/41308/p2p/16Uiu2HAm6sKWQgwCqahcDxwnTnpt3PMqp4hgmsBLJHwVtr6x26iJ","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-subscribe/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-13 04:32:36.730 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21843/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:32:36.733 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.158.229/tcp/33595/p2p/16Uiu2HAkz32XHRtUtc6xziZKLmUtKQx5Sn4fC8eFP3LKxPPNXTCw","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:32:36.733 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-03-13 04:33:06.733 DEBUG [tests.e2e.test_e2e] Edge node2 makes filter subscription to pubsubtopic /waku/2/rs/3/1 and content topic /test/1/waku-filter/proto
2026-03-13 04:33:06.734 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:56494/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "1", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-03-13 04:33:06.777 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1","statusDesc":"OK"}'
2026-03-13 04:33:06.777 DEBUG [tests.e2e.test_e2e] Check if service node1 receives message sent by edge node1
2026-03-13 04:33:06.778 DEBUG [src.steps.light_push] Lightpushing message
2026-03-13 04:33:06.781 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:40892/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/1", "message": {"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": 1773376386778325988}}'
2026-03-13 04:33:06.839 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:33:06.842 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-03-13 04:33:06.944 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-03-13 04:33:06.944 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65309/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:33:06.947 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1773376386778325988,"ephemeral":false,"proof":""}]'
2026-03-13 04:33:06.949 DEBUG [tests.e2e.test_e2e] Check if edge node2 can query stored message
2026-03-13 04:33:06.949 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-13 04:33:06.949 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56494/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:33:06.981 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"d3a04ffc4126589958ba","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x6d2fa74146aa4d2e98add6d5977c06ad7a9a4d923d46b0debe4c7735a0dfcb41"}]}'
2026-03-13 04:33:06.983 DEBUG [src.steps.store] messages length is 1
2026-03-13 04:33:06.983 DEBUG [tests.e2e.test_e2e] Check if service node3 can query stored message
2026-03-13 04:33:06.985 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-03-13 04:33:06.986 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21843/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:33:06.996 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"eb177092dd6e746a28a8","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0x6d2fa74146aa4d2e98add6d5977c06ad7a9a4d923d46b0debe4c7735a0dfcb41"}]}'
2026-03-13 04:33:06.996 DEBUG [src.steps.store] messages length is 1
2026-03-13 04:33:06.996 DEBUG [tests.e2e.test_e2e] Check if edge node2 can get sent message using filter get request
2026-03-13 04:33:06.997 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:56494/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:33:06.999 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1773376386778325988,"ephemeral":false}]'
2026-03-13 04:33:07.002 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:33:07.003 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:33:07.003 DEBUG [src.node.waku_node] Stopping container with id 23c6504139a7
2026-03-13 04:33:07.609 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:33:07.609 DEBUG [src.node.waku_node] Stopping container with id 337da2ea8484
2026-03-13 04:33:08.183 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:33:08.184 DEBUG [src.node.waku_node] Stopping container with id eff452bcb466
2026-03-13 04:33:08.422 ERROR [src.node.docker_mananger] Max retries reached for container 23c6504139a7. Exiting log stream.
2026-03-13 04:33:08.791 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:33:08.791 DEBUG [src.node.waku_node] Stopping container with id d27bc20e7d6f
2026-03-13 04:33:09.000 ERROR [src.node.docker_mananger] Max retries reached for container 337da2ea8484. Exiting log stream.
2026-03-13 04:33:09.419 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:33:09.419 DEBUG [src.node.waku_node] Stopping container with id d75fdd256e1e
2026-03-13 04:33:09.706 ERROR [src.node.docker_mananger] Max retries reached for container eff452bcb466. Exiting log stream.
2026-03-13 04:33:10.001 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:33:10.005 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:33:10.034 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:33:10.041 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:33:10.098 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:33:10.125 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:33:10.133 DEBUG [src.node.docker_mananger] No errors found in the waku logs.