159 lines
33 KiB
Plaintext

2026-03-26 04:46:13.298 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-26 04:46:13.298 DEBUG [tests.conftest] Running test: test_relay_4_nodes_two_publishers_compare_latency with id: 2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6
2026-03-26 04:46:13.299 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-26 04:46:13.299 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-26 04:46:13.306 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:13.306 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:13.312 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:13.312 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:13.319 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:13.319 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:13.325 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:13.325 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:13.326 INFO [tests.e2e.test_network_conditions] Starting 4 nodes
2026-03-26 04:46:13.326 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:13.326 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:13.327 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:13.327 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.248.40
2026-03-26 04:46:13.328 DEBUG [src.node.docker_mananger] Generated ports ['62716', '62717', '62718', '62719', '62720']
2026-03-26 04:46:13.328 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:13.328 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:13.328 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:13.328 DEBUG [src.node.docker_mananger] docker run -i -t -p 62716:62716 -p 62717:62717 -p 62718:62718 -p 62719:62719 -p 62720:62720 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62718 --rest-port=62716 --tcp-port=62717 --discv5-udp-port=62719 --rest-address=0.0.0.0 --nat=extip:172.18.248.40 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=09c1d31d9b2d96e41a3f1bead0fd0ca5f31a6bc8e0e50ecc69b1ee422a0f4cc1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62720 --metrics-logging=true --relay=true
2026-03-26 04:46:13.524 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.248.40 waku 93dbc4a7afde64fa39cbf69e220e0a7ce67cafa44298eb0169dbe39fbec72cbd
2026-03-26 04:46:13.543 ERROR [src.node.docker_mananger] Max retries reached for container 1db5c75cf396. Exiting log stream.
2026-03-26 04:46:13.558 DEBUG [src.node.docker_mananger] Container started with ID 93dbc4a7afde. Setting up logs at ./log/docker/node1_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:13.558 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 62716
2026-03-26 04:46:13.559 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:14.060 ERROR [src.node.docker_mananger] Max retries reached for container 94afa037ed4a. Exiting log stream.
2026-03-26 04:46:14.559 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62716/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:14.562 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:14.562 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:14.562 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62716/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:14.565 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.248.40/tcp/62717/p2p/16Uiu2HAmTAmWb97NqiUom2mt6QtfJhQCavNALCvxqQfi1o3LJ4vu","/ip4/172.18.248.40/tcp/62718/ws/p2p/16Uiu2HAmTAmWb97NqiUom2mt6QtfJhQCavNALCvxqQfi1o3LJ4vu"],"enrUri":"enr:-L24QNHE81w4LFnbMf8ajSEAtCMdvMBT-WoO1yJUjc7a-hr2NMMcGEMgpLBnzRIi98DkBq3Z6dPHt91unrswWIq7KycCgmlkgnY0gmlwhKwS-CiKbXVsdGlhZGRyc5YACASsEvgoBvT9AAoErBL4KAb0_t0DgnJzhQADAQAAiXNlY3AyNTZrMaED16SF_MST4YDxtyugAKP1EiFK8ALCFawHiSJyUZbf5DqDdGNwgvT9g3VkcIL0_4V3YWt1MgE"}'
2026-03-26 04:46:14.565 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:14.565 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:14.565 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:14.567 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:14.567 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.222.97
2026-03-26 04:46:14.567 DEBUG [src.node.docker_mananger] Generated ports ['10752', '10753', '10754', '10755', '10756']
2026-03-26 04:46:14.567 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:14.567 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:14.568 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:14.568 DEBUG [src.node.docker_mananger] docker run -i -t -p 10752:10752 -p 10753:10753 -p 10754:10754 -p 10755:10755 -p 10756:10756 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=10754 --rest-port=10752 --tcp-port=10753 --discv5-udp-port=10755 --rest-address=0.0.0.0 --nat=extip:172.18.222.97 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=eb02c934dcb9d9cea4cc4ae8c91eedb4f7c56d4b85c3795cfebcb41bc7e8d0d9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=10756 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QNHE81w4LFnbMf8ajSEAtCMdvMBT-WoO1yJUjc7a-hr2NMMcGEMgpLBnzRIi98DkBq3Z6dPHt91unrswWIq7KycCgmlkgnY0gmlwhKwS-CiKbXVsdGlhZGRyc5YACASsEvgoBvT9AAoErBL4KAb0_t0DgnJzhQADAQAAiXNlY3AyNTZrMaED16SF_MST4YDxtyugAKP1EiFK8ALCFawHiSJyUZbf5DqDdGNwgvT9g3VkcIL0_4V3YWt1MgE
2026-03-26 04:46:14.765 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.222.97 waku 086350a28ebdebd07ae8ac46d7520efa9ba578b7fd0db0f30e7a51da76b23595
2026-03-26 04:46:14.798 DEBUG [src.node.docker_mananger] Container started with ID 086350a28ebd. Setting up logs at ./log/docker/node2_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:14.799 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 10752
2026-03-26 04:46:14.799 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:15.800 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10752/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:15.821 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:15.824 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:15.824 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10752/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:15.830 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.222.97/tcp/10753/p2p/16Uiu2HAkygnxKqzrcJPqP86FvBuG9qjscgiqRg8gUe9c6gwuqetB","/ip4/172.18.222.97/tcp/10754/ws/p2p/16Uiu2HAkygnxKqzrcJPqP86FvBuG9qjscgiqRg8gUe9c6gwuqetB"],"enrUri":"enr:-L24QPmtX9RU3Av-wQbYVCJnK8N0PBJDN0S1QX2HPDJZWz40Py4LZiAidL5P1ZprVqwmPh4uIu9Iahs2aLmQ6qVOoy0CgmlkgnY0gmlwhKwS3mGKbXVsdGlhZGRyc5YACASsEt5hBioBAAoErBLeYQYqAt0DgnJzhQADAQAAiXNlY3AyNTZrMaECP05YnXDNUj1gRs1h6ItX0nesw_qTbZQ_3pieF7vAx4yDdGNwgioBg3VkcIIqA4V3YWt1MgE"}'
2026-03-26 04:46:15.831 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:15.831 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:15.831 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:15.834 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:15.834 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.253.182
2026-03-26 04:46:15.834 DEBUG [src.node.docker_mananger] Generated ports ['59206', '59207', '59208', '59209', '59210']
2026-03-26 04:46:15.834 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:15.834 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:15.835 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:15.835 DEBUG [src.node.docker_mananger] docker run -i -t -p 59206:59206 -p 59207:59207 -p 59208:59208 -p 59209:59209 -p 59210:59210 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=59208 --rest-port=59206 --tcp-port=59207 --discv5-udp-port=59209 --rest-address=0.0.0.0 --nat=extip:172.18.253.182 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=8a3f877d5fe8eecf7d72efdaf9fcff57ee40febd6f5fc99affedd2bf1a7ceef1 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=59210 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPmtX9RU3Av-wQbYVCJnK8N0PBJDN0S1QX2HPDJZWz40Py4LZiAidL5P1ZprVqwmPh4uIu9Iahs2aLmQ6qVOoy0CgmlkgnY0gmlwhKwS3mGKbXVsdGlhZGRyc5YACASsEt5hBioBAAoErBLeYQYqAt0DgnJzhQADAQAAiXNlY3AyNTZrMaECP05YnXDNUj1gRs1h6ItX0nesw_qTbZQ_3pieF7vAx4yDdGNwgioBg3VkcIIqA4V3YWt1MgE
2026-03-26 04:46:16.020 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.253.182 waku 6337582a36085f54cf0a444df529ea05c17c4783516f6cfcd7318a9aaf8dece2
2026-03-26 04:46:16.057 DEBUG [src.node.docker_mananger] Container started with ID 6337582a3608. Setting up logs at ./log/docker/node3_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:16.057 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 59206
2026-03-26 04:46:16.058 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:17.058 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59206/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:17.072 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:17.074 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:17.074 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59206/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:17.078 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.253.182/tcp/59207/p2p/16Uiu2HAm6rswcB6QbD1MHtHY1yWhYtmp7nz5qxsqBFb1GV7dniTs","/ip4/172.18.253.182/tcp/59208/ws/p2p/16Uiu2HAm6rswcB6QbD1MHtHY1yWhYtmp7nz5qxsqBFb1GV7dniTs"],"enrUri":"enr:-L24QKhzSocswGTSoWY203XmoYkqQszczPkkP2oXFAXhkf-7BxGjG9kw4dJndF-xQewCtCuC2o7bbsXxWwN2XgjKB6ECgmlkgnY0gmlwhKwS_baKbXVsdGlhZGRyc5YACASsEv22BudHAAoErBL9tgbnSN0DgnJzhQADAQAAiXNlY3AyNTZrMaECqeWaLFpGr70po_n-3-_IZ9Dx6YLI0daeNSuAjZKGmmKDdGNwgudHg3VkcILnSYV3YWt1MgE"}'
2026-03-26 04:46:17.080 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:17.082 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:17.082 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:17.085 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:17.086 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.177.204
2026-03-26 04:46:17.086 DEBUG [src.node.docker_mananger] Generated ports ['33702', '33703', '33704', '33705', '33706']
2026-03-26 04:46:17.086 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:17.086 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:17.086 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:17.086 DEBUG [src.node.docker_mananger] docker run -i -t -p 33702:33702 -p 33703:33703 -p 33704:33704 -p 33705:33705 -p 33706:33706 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=33704 --rest-port=33702 --tcp-port=33703 --discv5-udp-port=33705 --rest-address=0.0.0.0 --nat=extip:172.18.177.204 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f92e3d9b3fda653a8cb492aadaacb3f9839cf105f7e19ff8a1aa74ca9b6ead26 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=33706 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QKhzSocswGTSoWY203XmoYkqQszczPkkP2oXFAXhkf-7BxGjG9kw4dJndF-xQewCtCuC2o7bbsXxWwN2XgjKB6ECgmlkgnY0gmlwhKwS_baKbXVsdGlhZGRyc5YACASsEv22BudHAAoErBL9tgbnSN0DgnJzhQADAQAAiXNlY3AyNTZrMaECqeWaLFpGr70po_n-3-_IZ9Dx6YLI0daeNSuAjZKGmmKDdGNwgudHg3VkcILnSYV3YWt1MgE
2026-03-26 04:46:17.281 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.177.204 waku bb1c0304d074f49c91043191bb5c2dd18ff8e9eef98b08fce9eaa3d49c720304
2026-03-26 04:46:17.316 DEBUG [src.node.docker_mananger] Container started with ID bb1c0304d074. Setting up logs at ./log/docker/node4_2026-03-26_04-46-13__480637dc-bcf8-4f1e-a2b1-17c184facbd6__wakuorg_nwaku:latest.log
2026-03-26 04:46:17.316 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 33702
2026-03-26 04:46:17.316 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:18.318 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33702/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:18.339 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:18.342 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:18.344 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33702/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:18.347 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.177.204/tcp/33703/p2p/16Uiu2HAkuhSzBBjpvVgpiEkuj6GNhJaDCii2RddkyFtvkJxUaFL5","/ip4/172.18.177.204/tcp/33704/ws/p2p/16Uiu2HAkuhSzBBjpvVgpiEkuj6GNhJaDCii2RddkyFtvkJxUaFL5"],"enrUri":"enr:-L24QFA13SAu76CUV_opvm9KiMZh0BQgn8JmPIGdl9YuJPt6KSzLUYfShsgU8PvNO-pNfkE8eSO9Dsuk1zvGxQFDSUwCgmlkgnY0gmlwhKwSscyKbXVsdGlhZGRyc5YACASsErHMBoOnAAoErBKxzAaDqN0DgnJzhQADAQAAiXNlY3AyNTZrMaECBAqZ7F_TT8vdBZ6kE5GuQQBXsbjQAkRuOaSxUMqkekKDdGNwgoOng3VkcIKDqYV3YWt1MgE"}'
2026-03-26 04:46:18.347 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:18.347 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62716/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:46:18.351 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:46:18.351 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10752/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:46:18.358 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:46:18.360 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:59206/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:46:18.372 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:46:18.372 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:33702/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:46:18.379 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:46:18.380 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62716/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:18.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.222.97/tcp/55124/p2p/16Uiu2HAkygnxKqzrcJPqP86FvBuG9qjscgiqRg8gUe9c6gwuqetB","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-26 04:46:18.384 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10752/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:18.387 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.248.40/tcp/62717/p2p/16Uiu2HAmTAmWb97NqiUom2mt6QtfJhQCavNALCvxqQfi1o3LJ4vu","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.253.182/tcp/35554/p2p/16Uiu2HAm6rswcB6QbD1MHtHY1yWhYtmp7nz5qxsqBFb1GV7dniTs","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-26 04:46:18.387 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:59206/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:18.390 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.177.204/tcp/58824/p2p/16Uiu2HAkuhSzBBjpvVgpiEkuj6GNhJaDCii2RddkyFtvkJxUaFL5","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.222.97/tcp/10753/p2p/16Uiu2HAkygnxKqzrcJPqP86FvBuG9qjscgiqRg8gUe9c6gwuqetB","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-26 04:46:18.390 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33702/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:18.392 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.253.182/tcp/59207/p2p/16Uiu2HAm6rswcB6QbD1MHtHY1yWhYtmp7nz5qxsqBFb1GV7dniTs","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-26 04:46:18.393 DEBUG [src.libs.common] Sleeping for 60 seconds
2026-03-26 04:47:18.393 DEBUG [tests.e2e.test_network_conditions] Applying 3000ms latency on node1 only
2026-03-26 04:47:18.395 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '5032', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-26 04:47:18.457 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '5032', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'delay', '3000ms']
2026-03-26 04:47:18.471 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33702/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:18.474 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-26 04:47:18.475 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62716/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:24.484 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:24.484 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10752/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:24.495 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:24.495 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-26 04:47:24.695 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62716/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:30.702 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:30.702 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10752/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:30.710 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:30.713 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-26 04:47:30.913 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62716/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:36.920 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:36.920 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10752/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:36.928 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:36.929 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-26 04:47:37.133 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62716/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:43.140 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:43.140 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10752/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:43.147 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:43.147 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-26 04:47:43.348 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62716/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:49.355 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:49.355 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10752/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-26 04:47:49.367 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:49.369 DEBUG [src.libs.common] Sleeping for 0.2 seconds
2026-03-26 04:47:49.572 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:33702/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:49.575 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500438474960256,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500444484597577,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500444695810810,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500450702663223,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500450913615373,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500456920530770,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500457133622128,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500463140686753,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500463348187380,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774500469355439719,"ephemeral":false,"proof":""}]'
2026-03-26 04:47:49.576 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '5032', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-26 04:47:49.588 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-26 04:47:49.589 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-26 04:47:49.589 DEBUG [src.node.waku_node] Stopping container with id 93dbc4a7afde
2026-03-26 04:47:50.175 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:47:50.176 DEBUG [src.node.waku_node] Stopping container with id 086350a28ebd
2026-03-26 04:47:50.768 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:47:50.772 DEBUG [src.node.waku_node] Stopping container with id 6337582a3608
2026-03-26 04:47:50.999 ERROR [src.node.docker_mananger] Max retries reached for container 93dbc4a7afde. Exiting log stream.
2026-03-26 04:47:51.327 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:47:51.328 DEBUG [src.node.waku_node] Stopping container with id bb1c0304d074
2026-03-26 04:47:51.865 ERROR [src.node.docker_mananger] Max retries reached for container 086350a28ebd. Exiting log stream.
2026-03-26 04:47:51.892 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:47:51.893 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-26 04:47:51.975 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:47:52.124 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:47:52.270 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:47:52.342 DEBUG [src.node.docker_mananger] No errors found in the waku logs.