179 lines
34 KiB
Plaintext

2026-01-29 04:30:45.955 DEBUG [tests.conftest] Running fixture setup: test_id
2026-01-29 04:30:45.956 DEBUG [tests.conftest] Running test: test_multiple_edge_service_nodes_communication with id: 2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6
2026-01-29 04:30:45.956 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-01-29 04:30:45.957 DEBUG [src.steps.filter] Running fixture setup: filter_setup
2026-01-29 04:30:45.957 DEBUG [src.steps.light_push] Running fixture setup: light_push_setup
2026-01-29 04:30:45.966 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:45.967 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:45.973 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:45.973 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:45.980 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:45.981 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:45.981 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-01-29 04:30:45.981 DEBUG [src.steps.store] Running fixture setup: store_setup
2026-01-29 04:30:45.989 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:45.989 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:45.996 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:45.996 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:46.003 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:46.003 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:46.010 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:46.010 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:46.017 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-01-29 04:30:46.017 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node8_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:46.017 DEBUG [tests.e2e.test_e2e] Start 2 edges nodes and 3 service nodes
2026-01-29 04:30:46.017 DEBUG [src.node.waku_node] Starting Node...
2026-01-29 04:30:46.017 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-29 04:30:46.060 DEBUG [src.node.docker_mananger] Network waku created
2026-01-29 04:30:46.060 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.98.146
2026-01-29 04:30:46.061 DEBUG [src.node.docker_mananger] Generated ports ['25249', '25250', '25251', '25252', '25253']
2026-01-29 04:30:46.061 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-29 04:30:46.061 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-29 04:30:46.061 DEBUG [src.node.waku_node] Using volumes []
2026-01-29 04:30:46.061 DEBUG [src.node.docker_mananger] docker run -i -t -p 25249:25249 -p 25250:25250 -p 25251:25251 -p 25252:25252 -p 25253:25253 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=25251 --rest-port=25249 --tcp-port=25250 --discv5-udp-port=25252 --rest-address=0.0.0.0 --nat=extip:172.18.98.146 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=520c3216ab1cfcfbdf1e1e65cc17e1b0c6b30da2bf6a845b1d4a6538d3ae688b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=25253 --metrics-logging=true --relay=true --store=true --lightpush=true
2026-01-29 04:30:48.869 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.98.146 waku 46ea5f9f0f5c3ff2c6e21878f119bb6f395a0cebc6a0faa838b3cd2055089b56
2026-01-29 04:30:48.900 DEBUG [src.node.docker_mananger] Container started with ID 46ea5f9f0f5c. Setting up logs at ./log/docker/node6_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:48.902 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 25249
2026-01-29 04:30:48.902 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-29 04:30:49.903 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25249/health" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:49.906 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Legacy Lightpush":"NOT_READY","desc":"Node has no relay peers to fullfill push requests"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2026-01-29 04:30:49.907 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-29 04:30:49.907 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25249/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:49.910 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.98.146/tcp/25250/p2p/16Uiu2HAkyhSMQJW5hnKm1r7251rrba9HGF7jJURyBX8R1Sadvi3Z","/ip4/172.18.98.146/tcp/25251/ws/p2p/16Uiu2HAkyhSMQJW5hnKm1r7251rrba9HGF7jJURyBX8R1Sadvi3Z"],"enrUri":"enr:-L24QM9FgYMrJUULFbzBZo2DRlt8siU-NlKcXuH22S5CwAZ9JgrVuyVqAk4q8ybGlElV8mVAUUKTeKZnVzhzR-qdP4oCgmlkgnY0gmlwhKwSYpKKbXVsdGlhZGRyc5YACASsEmKSBmKiAAoErBJikgZio90DgnJzhQADAQAAiXNlY3AyNTZrMaECP3ihzxg0YHMBl68HupWXYtXlBZE19UI0tcrrR6nu18CDdGNwgmKig3VkcIJipIV3YWt1Mgs"}'
2026-01-29 04:30:49.910 INFO [src.node.waku_node] REST service is ready !!
2026-01-29 04:30:49.910 DEBUG [src.node.waku_node] Starting Node...
2026-01-29 04:30:49.910 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-29 04:30:49.912 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-29 04:30:49.912 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.132.31
2026-01-29 04:30:49.912 DEBUG [src.node.docker_mananger] Generated ports ['35687', '35688', '35689', '35690', '35691']
2026-01-29 04:30:49.912 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-29 04:30:49.912 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-29 04:30:49.913 DEBUG [src.node.waku_node] Using volumes []
2026-01-29 04:30:49.913 DEBUG [src.node.docker_mananger] docker run -i -t -p 35687:35687 -p 35688:35688 -p 35689:35689 -p 35690:35690 -p 35691:35691 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=35689 --rest-port=35687 --tcp-port=35688 --discv5-udp-port=35690 --rest-address=0.0.0.0 --nat=extip:172.18.132.31 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=89cde0e543e0bf4850681aeff95db06e35bcaa7bf349d8496f92e9aa64c05e06 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=35691 --metrics-logging=true --relay=false --lightpushnode=/ip4/172.18.98.146/tcp/25250/p2p/16Uiu2HAkyhSMQJW5hnKm1r7251rrba9HGF7jJURyBX8R1Sadvi3Z --discv5-bootstrap-node=enr:-L24QM9FgYMrJUULFbzBZo2DRlt8siU-NlKcXuH22S5CwAZ9JgrVuyVqAk4q8ybGlElV8mVAUUKTeKZnVzhzR-qdP4oCgmlkgnY0gmlwhKwSYpKKbXVsdGlhZGRyc5YACASsEmKSBmKiAAoErBJikgZio90DgnJzhQADAQAAiXNlY3AyNTZrMaECP3ihzxg0YHMBl68HupWXYtXlBZE19UI0tcrrR6nu18CDdGNwgmKig3VkcIJipIV3YWt1Mgs
2026-01-29 04:30:50.101 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.132.31 waku 4e3d29342f54c10e159b17ccc03668b176c1c874946d0c0fc19712c63b02d29b
2026-01-29 04:30:50.134 DEBUG [src.node.docker_mananger] Container started with ID 4e3d29342f54. Setting up logs at ./log/docker/node4_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:50.134 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 35687
2026-01-29 04:30:50.135 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-29 04:30:51.136 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35687/health" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:51.140 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"READY"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2026-01-29 04:30:51.140 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-29 04:30:51.141 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:35687/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:51.143 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.132.31/tcp/35688/p2p/16Uiu2HAkvLSMLrkyPhwndDddackwqzyUZRtUbEP3hR6yByGXUXGC","/ip4/172.18.132.31/tcp/35689/ws/p2p/16Uiu2HAkvLSMLrkyPhwndDddackwqzyUZRtUbEP3hR6yByGXUXGC"],"enrUri":"enr:-L24QEQm9AFdyJbdJWcqDEXRt0BoLEw50OeCSwaEaJxeU33kSmDzjCKkkIN-jdm8SqhGHDu9LB_EkgZIJRyBzdmBub0CgmlkgnY0gmlwhKwShB-KbXVsdGlhZGRyc5YACASsEoQfBotoAAoErBKEHwaLad0DgnJzhQADAQAAiXNlY3AyNTZrMaECDYRfhgsN5nvwaklNSNvG8RWMMmSe5_c6VA2E-0TNvSGDdGNwgotog3VkcIKLaoV3YWt1MgA"}'
2026-01-29 04:30:51.143 INFO [src.node.waku_node] REST service is ready !!
2026-01-29 04:30:51.143 DEBUG [src.node.waku_node] Starting Node...
2026-01-29 04:30:51.144 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-29 04:30:51.145 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-29 04:30:51.145 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.77.93
2026-01-29 04:30:51.145 DEBUG [src.node.docker_mananger] Generated ports ['6113', '6114', '6115', '6116', '6117']
2026-01-29 04:30:51.146 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-29 04:30:51.146 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-29 04:30:51.146 DEBUG [src.node.waku_node] Using volumes []
2026-01-29 04:30:51.146 DEBUG [src.node.docker_mananger] docker run -i -t -p 6113:6113 -p 6114:6114 -p 6115:6115 -p 6116:6116 -p 6117:6117 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=6115 --rest-port=6113 --tcp-port=6114 --discv5-udp-port=6116 --rest-address=0.0.0.0 --nat=extip:172.18.77.93 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=eaf34dadff2e145820c7f54b6cd99b3916fbd460f98a9a16f1b8f8bb1f912d0f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=6117 --metrics-logging=true --relay=true --store=true --discv5-bootstrap-node=enr:-L24QM9FgYMrJUULFbzBZo2DRlt8siU-NlKcXuH22S5CwAZ9JgrVuyVqAk4q8ybGlElV8mVAUUKTeKZnVzhzR-qdP4oCgmlkgnY0gmlwhKwSYpKKbXVsdGlhZGRyc5YACASsEmKSBmKiAAoErBJikgZio90DgnJzhQADAQAAiXNlY3AyNTZrMaECP3ihzxg0YHMBl68HupWXYtXlBZE19UI0tcrrR6nu18CDdGNwgmKig3VkcIJipIV3YWt1Mgs
2026-01-29 04:30:51.359 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.77.93 waku a13a274d915baf91285aa8ba522937a28ee3b30602f1f882bc961e59b81bff9b
2026-01-29 04:30:51.392 DEBUG [src.node.docker_mananger] Container started with ID a13a274d915b. Setting up logs at ./log/docker/node7_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:51.394 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 6113
2026-01-29 04:30:51.394 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-29 04:30:52.394 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6113/health" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:52.398 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"READY"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"READY"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2026-01-29 04:30:52.399 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-29 04:30:52.399 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6113/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:52.401 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.77.93/tcp/6114/p2p/16Uiu2HAkxGTN5PVR57YXkPwfeYBNzd8ag4vGuyfV6ii6LUb6rtBb","/ip4/172.18.77.93/tcp/6115/ws/p2p/16Uiu2HAkxGTN5PVR57YXkPwfeYBNzd8ag4vGuyfV6ii6LUb6rtBb"],"enrUri":"enr:-L24QMHSZUUlFFxqxyUo-QqcFkpPITMNJvN875zTHnOKY4IyPsRDiPCTQUNo3NVMKpLz_BywYpub-Dfe15lr2vXp7JwCgmlkgnY0gmlwhKwSTV2KbXVsdGlhZGRyc5YACASsEk1dBhfiAAoErBJNXQYX490DgnJzhQADAQAAiXNlY3AyNTZrMaECKjaRmd-8CwsRw3-y4Djih_fe53nfNjZIPbgbGd4V7gqDdGNwghfig3VkcIIX5IV3YWt1MgM"}'
2026-01-29 04:30:52.401 INFO [src.node.waku_node] REST service is ready !!
2026-01-29 04:30:52.401 DEBUG [src.node.waku_node] Starting Node...
2026-01-29 04:30:52.402 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-29 04:30:52.403 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-29 04:30:52.403 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.122.252
2026-01-29 04:30:52.403 DEBUG [src.node.docker_mananger] Generated ports ['65503', '65504', '65505', '65506', '65507']
2026-01-29 04:30:52.404 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-29 04:30:52.404 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-29 04:30:52.404 DEBUG [src.node.waku_node] Using volumes []
2026-01-29 04:30:52.404 DEBUG [src.node.docker_mananger] docker run -i -t -p 65503:65503 -p 65504:65504 -p 65505:65505 -p 65506:65506 -p 65507:65507 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=65505 --rest-port=65503 --tcp-port=65504 --discv5-udp-port=65506 --rest-address=0.0.0.0 --nat=extip:172.18.122.252 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=4b454338d3eeb9adcf59ab61e74d3d3c8fedfc0d4a105cceacfefbb6b3d3ae4c --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=65507 --metrics-logging=true --relay=true --filter=true --storenode=/ip4/172.18.77.93/tcp/6114/p2p/16Uiu2HAkxGTN5PVR57YXkPwfeYBNzd8ag4vGuyfV6ii6LUb6rtBb --discv5-bootstrap-node=enr:-L24QMHSZUUlFFxqxyUo-QqcFkpPITMNJvN875zTHnOKY4IyPsRDiPCTQUNo3NVMKpLz_BywYpub-Dfe15lr2vXp7JwCgmlkgnY0gmlwhKwSTV2KbXVsdGlhZGRyc5YACASsEk1dBhfiAAoErBJNXQYX490DgnJzhQADAQAAiXNlY3AyNTZrMaECKjaRmd-8CwsRw3-y4Djih_fe53nfNjZIPbgbGd4V7gqDdGNwghfig3VkcIIX5IV3YWt1MgM
2026-01-29 04:30:52.636 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.122.252 waku 584afb16a45333e7dd86214135ce8490e5908bc3753685f75bcefe081837b369
2026-01-29 04:30:52.667 DEBUG [src.node.docker_mananger] Container started with ID 584afb16a453. Setting up logs at ./log/docker/node8_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:52.668 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 65503
2026-01-29 04:30:52.668 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-29 04:30:53.669 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65503/health" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:53.673 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_READY","desc":"Relay is not ready, filter will not be able to sort out messages"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"READY"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"}]}'
2026-01-29 04:30:53.673 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-29 04:30:53.673 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65503/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:53.675 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.122.252/tcp/65504/p2p/16Uiu2HAm7DfJ8LsGp54fGKMqgnVXVKNn6Rq6JStHa7VcwQNXpGa6","/ip4/172.18.122.252/tcp/65505/ws/p2p/16Uiu2HAm7DfJ8LsGp54fGKMqgnVXVKNn6Rq6JStHa7VcwQNXpGa6"],"enrUri":"enr:-L24QPaK5IAOA9-A7FBmQXuBbXKPFQ-dayqLJ3CjkhaxvdzZYgjxqxmvJtA_gUWadQJH9m6eMe23LpQdAFXFowtL3hcCgmlkgnY0gmlwhKwSevyKbXVsdGlhZGRyc5YACASsEnr8Bv_gAAoErBJ6_Ab_4d0DgnJzhQADAQAAiXNlY3AyNTZrMaECrziAB-2yl69pncpwwLZ7swLHtkjJjbmUD6g-fmYgybODdGNwgv_gg3VkcIL_4oV3YWt1MgU"}'
2026-01-29 04:30:53.676 INFO [src.node.waku_node] REST service is ready !!
2026-01-29 04:30:53.676 DEBUG [src.node.waku_node] Starting Node...
2026-01-29 04:30:53.676 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-01-29 04:30:53.677 DEBUG [src.node.docker_mananger] Network waku already exists
2026-01-29 04:30:53.678 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.64.75
2026-01-29 04:30:53.678 DEBUG [src.node.docker_mananger] Generated ports ['53175', '53176', '53177', '53178', '53179']
2026-01-29 04:30:53.678 DEBUG [src.node.waku_node] RLN credentials were not set
2026-01-29 04:30:53.678 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-01-29 04:30:53.678 DEBUG [src.node.waku_node] Using volumes []
2026-01-29 04:30:53.679 DEBUG [src.node.docker_mananger] docker run -i -t -p 53175:53175 -p 53176:53176 -p 53177:53177 -p 53178:53178 -p 53179:53179 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=53177 --rest-port=53175 --tcp-port=53176 --discv5-udp-port=53178 --rest-address=0.0.0.0 --nat=extip:172.18.64.75 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=6b9c7db6ccee098b1b78acdac3892ceec5a8ae33cc26fee8332f4cd7eadcc10f --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=53179 --metrics-logging=true --relay=false --filternode=/ip4/172.18.122.252/tcp/65504/p2p/16Uiu2HAm7DfJ8LsGp54fGKMqgnVXVKNn6Rq6JStHa7VcwQNXpGa6 --storenode=/ip4/172.18.77.93/tcp/6114/p2p/16Uiu2HAkxGTN5PVR57YXkPwfeYBNzd8ag4vGuyfV6ii6LUb6rtBb --discv5-bootstrap-node=enr:-L24QMHSZUUlFFxqxyUo-QqcFkpPITMNJvN875zTHnOKY4IyPsRDiPCTQUNo3NVMKpLz_BywYpub-Dfe15lr2vXp7JwCgmlkgnY0gmlwhKwSTV2KbXVsdGlhZGRyc5YACASsEk1dBhfiAAoErBJNXQYX490DgnJzhQADAQAAiXNlY3AyNTZrMaECKjaRmd-8CwsRw3-y4Djih_fe53nfNjZIPbgbGd4V7gqDdGNwghfig3VkcIIX5IV3YWt1MgM
2026-01-29 04:30:53.890 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.64.75 waku b1bf6d7fb62f935f30f269901a4d22ad62329de6863904e24b472a4218d9a7d1
2026-01-29 04:30:53.921 DEBUG [src.node.docker_mananger] Container started with ID b1bf6d7fb62f. Setting up logs at ./log/docker/node5_2026-01-29_04-30-45__ebee3d7e-3686-453f-922b-2a0659a662b6__wakuorg_nwaku:latest.log
2026-01-29 04:30:53.921 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 53175
2026-01-29 04:30:53.921 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-01-29 04:30:54.923 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:53175/health" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:54.927 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","protocolsHealth":[{"Relay":"NOT_MOUNTED"},{"Rln Relay":"NOT_MOUNTED"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"READY"},{"Legacy Store Client":"READY"},{"Filter Client":"READY"}]}'
2026-01-29 04:30:54.927 INFO [src.node.waku_node] Node protocols are initialized !!
2026-01-29 04:30:54.927 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:53175/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:54.929 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.64.75/tcp/53176/p2p/16Uiu2HAmL25Hj6wp5LHbMnALvTRDrJkPhvWRhoxmNRvfd2wky5NG","/ip4/172.18.64.75/tcp/53177/ws/p2p/16Uiu2HAmL25Hj6wp5LHbMnALvTRDrJkPhvWRhoxmNRvfd2wky5NG"],"enrUri":"enr:-L24QIDXNA6mll42BE1EgfCxFZUWtEsis4hfkaHgqrmQmModEOye4EY5AOWsrvBrnCfwXWENI-Mp6EbDPVbV2rtHbk4CgmlkgnY0gmlwhKwSQEuKbXVsdGlhZGRyc5YACASsEkBLBs-4AAoErBJASwbPud0DgnJzhQADAQAAiXNlY3AyNTZrMaEDbWiW62wrkfy1N--Zue-UEvjHkRitrBwTBFtjK7ev49GDdGNwgs-4g3VkcILPuoV3YWt1MgA"}'
2026-01-29 04:30:54.930 INFO [src.node.waku_node] REST service is ready !!
2026-01-29 04:30:54.930 DEBUG [tests.e2e.test_e2e] Connect 3 service nodes to relay subscriptions
2026-01-29 04:30:54.930 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:25249/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-01-29 04:30:54.934 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-29 04:30:54.935 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:6113/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-01-29 04:30:54.942 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-29 04:30:54.943 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:65503/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-01-29 04:30:54.949 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-29 04:30:54.949 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25249/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:54.952 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.77.93/tcp/37628/p2p/16Uiu2HAkxGTN5PVR57YXkPwfeYBNzd8ag4vGuyfV6ii6LUb6rtBb","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-127-ga4e44d","origin":"Discv5"}]'
2026-01-29 04:30:54.952 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:6113/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:54.955 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.17.0.5/tcp/33094/p2p/16Uiu2HAm7DfJ8LsGp54fGKMqgnVXVKNn6Rq6JStHa7VcwQNXpGa6","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-subscribe/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-127-ga4e44d","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.98.146/tcp/25250/p2p/16Uiu2HAkyhSMQJW5hnKm1r7251rrba9HGF7jJURyBX8R1Sadvi3Z","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/lightpush/3.0.0","/vac/waku/lightpush/2.0.0-beta1","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-127-ga4e44d","origin":"Discv5"}]'
2026-01-29 04:30:54.955 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65503/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:30:54.957 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.77.93/tcp/6114/p2p/16Uiu2HAkxGTN5PVR57YXkPwfeYBNzd8ag4vGuyfV6ii6LUb6rtBb","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/store-query/3.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.36.0-127-ga4e44d","origin":"Discv5"}]'
2026-01-29 04:30:54.958 DEBUG [src.libs.common] Sleeping for 30 seconds
2026-01-29 04:31:24.958 DEBUG [tests.e2e.test_e2e] Edge node2 makes filter subscription to pubsubtopic /waku/2/rs/3/1 and content topic /test/1/waku-filter/proto
2026-01-29 04:31:24.958 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:53175/filter/v2/subscriptions" -H "Content-Type: application/json" -d '{"requestId": "1", "contentFilters": ["/test/1/waku-filter/proto"], "pubsubTopic": "/waku/2/rs/3/1"}'
2026-01-29 04:31:25.000 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"1","statusDesc":"OK"}'
2026-01-29 04:31:25.001 DEBUG [tests.e2e.test_e2e] Check if service node1 receives message sent by edge node1
2026-01-29 04:31:25.002 DEBUG [src.steps.light_push] Lightpushing message
2026-01-29 04:31:25.003 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:35687/lightpush/v1/message" -H "Content-Type: application/json" -d '{"pubsubTopic": "/waku/2/rs/3/1", "message": {"payload": "RmlsdGVyIHdvcmtzISE=", "contentTopic": "/test/1/waku-filter/proto", "timestamp": 1769661085002434425}}'
2026-01-29 04:31:25.050 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-01-29 04:31:25.053 DEBUG [src.libs.common] Sleeping for 0.1 seconds
2026-01-29 04:31:25.156 DEBUG [src.steps.light_push] Checking that peer NODE_1:wakuorg/nwaku:latest can find the lightpushed message
2026-01-29 04:31:25.156 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:25249/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:31:25.159 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1769661085002434425,"ephemeral":false,"proof":""}]'
2026-01-29 04:31:25.161 DEBUG [tests.e2e.test_e2e] Check if edge node2 can query stored message
2026-01-29 04:31:25.161 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-01-29 04:31:25.162 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:53175/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:31:25.191 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"f987d85c04e257e25770","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xdeddd95a257bd1a90aaa92fb522e1e31e3cb1c3f63be6f88d60c03e325fdc744"}]}'
2026-01-29 04:31:25.192 DEBUG [src.steps.store] messages length is 1
2026-01-29 04:31:25.193 DEBUG [tests.e2e.test_e2e] Check if service node3 can query stored message
2026-01-29 04:31:25.193 DEBUG [src.steps.store] Checking that peer wakuorg/nwaku:latest can find the stored messages
2026-01-29 04:31:25.193 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:65503/store/v3/messages?pubsubTopic=%2Fwaku%2F2%2Frs%2F3%2F1&pageSize=50&ascending=true" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:31:25.203 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"requestId":"4e963b9223effd792a7b","statusCode":200,"statusDesc":"OK","messages":[{"messageHash":"0xdeddd95a257bd1a90aaa92fb522e1e31e3cb1c3f63be6f88d60c03e325fdc744"}]}'
2026-01-29 04:31:25.203 DEBUG [src.steps.store] messages length is 1
2026-01-29 04:31:25.203 DEBUG [tests.e2e.test_e2e] Check if edge node2 can get sent message using filter get request
2026-01-29 04:31:25.204 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:53175/filter/v2/messages/%2Ftest%2F1%2Fwaku-filter%2Fproto" -H "Content-Type: application/json" -d 'None'
2026-01-29 04:31:25.211 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"RmlsdGVyIHdvcmtzISE=","contentTopic":"/test/1/waku-filter/proto","version":0,"timestamp":1769661085002434425,"ephemeral":false}]'
2026-01-29 04:31:25.216 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-01-29 04:31:25.217 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-01-29 04:31:25.217 DEBUG [src.node.waku_node] Stopping container with id 46ea5f9f0f5c
2026-01-29 04:31:25.798 DEBUG [src.node.waku_node] Container stopped.
2026-01-29 04:31:25.799 DEBUG [src.node.waku_node] Stopping container with id 4e3d29342f54
2026-01-29 04:31:26.408 DEBUG [src.node.waku_node] Container stopped.
2026-01-29 04:31:26.413 DEBUG [src.node.waku_node] Stopping container with id a13a274d915b
2026-01-29 04:31:26.619 ERROR [src.node.docker_mananger] Max retries reached for container 46ea5f9f0f5c. Exiting log stream.
2026-01-29 04:31:27.003 DEBUG [src.node.waku_node] Container stopped.
2026-01-29 04:31:27.004 DEBUG [src.node.waku_node] Stopping container with id 584afb16a453
2026-01-29 04:31:27.227 ERROR [src.node.docker_mananger] Max retries reached for container 4e3d29342f54. Exiting log stream.
2026-01-29 04:31:27.543 DEBUG [src.node.waku_node] Container stopped.
2026-01-29 04:31:27.543 DEBUG [src.node.waku_node] Stopping container with id b1bf6d7fb62f
2026-01-29 04:31:27.817 ERROR [src.node.docker_mananger] Max retries reached for container a13a274d915b. Exiting log stream.
2026-01-29 04:31:28.078 DEBUG [src.node.waku_node] Container stopped.
2026-01-29 04:31:28.079 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-01-29 04:31:28.107 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-29 04:31:28.113 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-29 04:31:28.139 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-29 04:31:28.191 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-01-29 04:31:28.198 DEBUG [src.node.docker_mananger] No errors found in the waku logs.