226 lines
47 KiB
Plaintext

2026-03-09 04:35:54.688 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-09 04:35:54.688 DEBUG [tests.conftest] Running test: test_relay_7_nodes_3sec_latency with id: 2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc
2026-03-09 04:35:54.689 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-09 04:35:54.689 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-09 04:35:54.695 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.695 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.701 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.702 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.708 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.708 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.714 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.714 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.720 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.720 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.726 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.726 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.732 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.732 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.738 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.738 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.744 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-09 04:35:54.744 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.744 INFO [tests.e2e.test_network_conditions] Starting nodes with chain bootstrap
2026-03-09 04:35:54.744 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:35:54.744 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:35:54.746 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:35:54.746 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.142.34
2026-03-09 04:35:54.746 DEBUG [src.node.docker_mananger] Generated ports ['44042', '44043', '44044', '44045', '44046']
2026-03-09 04:35:54.746 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:35:54.746 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:35:54.746 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:35:54.746 DEBUG [src.node.docker_mananger] docker run -i -t -p 44042:44042 -p 44043:44043 -p 44044:44044 -p 44045:44045 -p 44046:44046 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=44044 --rest-port=44042 --tcp-port=44043 --discv5-udp-port=44045 --rest-address=0.0.0.0 --nat=extip:172.18.142.34 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=dbea7b4da34b693f4d7acfe4f1eacfc6ade1498ff66286e9fca9ce4a8da5b33d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=44046 --metrics-logging=true --relay=true
2026-03-09 04:35:54.887 ERROR [src.node.docker_mananger] Max retries reached for container 91c4234f2238. Exiting log stream.
2026-03-09 04:35:54.947 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.142.34 waku 3cb9fe97dd6d5fc962468e5ccf940f9533abbe901fb328ad7e706aff1cdea534
2026-03-09 04:35:54.980 DEBUG [src.node.docker_mananger] Container started with ID 3cb9fe97dd6d. Setting up logs at ./log/docker/node1_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:54.981 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 44042
2026-03-09 04:35:54.981 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:35:55.450 ERROR [src.node.docker_mananger] Max retries reached for container 86d6e3888a0e. Exiting log stream.
2026-03-09 04:35:55.982 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44042/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:55.985 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:35:55.985 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:35:55.985 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44042/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:55.988 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.142.34/tcp/44043/p2p/16Uiu2HAmEkc3A1V4EtLN7azmYoBbys4pcGyxbE79oLvRxEyND2oM","/ip4/172.18.142.34/tcp/44044/ws/p2p/16Uiu2HAmEkc3A1V4EtLN7azmYoBbys4pcGyxbE79oLvRxEyND2oM"],"enrUri":"enr:-L24QP_PD0zkyS4YVvV5FCIownDvR1b87pHCPrueQ_gWCBUtL9DFSJutZmAPMzoIxGS1JoOriKnhTyHzk-BCPr_bfzYCgmlkgnY0gmlwhKwSjiKKbXVsdGlhZGRyc5YACASsEo4iBqwLAAoErBKOIgasDN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDHyehNNb_eEw6-O-XMpFaJ4LE5kVF1RYT6Eif4dj4h5SDdGNwgqwLg3VkcIKsDYV3YWt1MgE"}'
2026-03-09 04:35:55.988 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:35:55.988 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:35:55.988 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:35:55.990 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:35:55.990 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.47.128
2026-03-09 04:35:55.990 DEBUG [src.node.docker_mananger] Generated ports ['36917', '36918', '36919', '36920', '36921']
2026-03-09 04:35:55.990 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:35:55.990 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:35:55.990 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:35:55.991 DEBUG [src.node.docker_mananger] docker run -i -t -p 36917:36917 -p 36918:36918 -p 36919:36919 -p 36920:36920 -p 36921:36921 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=36919 --rest-port=36917 --tcp-port=36918 --discv5-udp-port=36920 --rest-address=0.0.0.0 --nat=extip:172.18.47.128 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=eeda12a0c1abb342de44ecf3ac15844ca1a9a3b5f420fa3b2c7b7dd8ffe08b47 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=36921 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QP_PD0zkyS4YVvV5FCIownDvR1b87pHCPrueQ_gWCBUtL9DFSJutZmAPMzoIxGS1JoOriKnhTyHzk-BCPr_bfzYCgmlkgnY0gmlwhKwSjiKKbXVsdGlhZGRyc5YACASsEo4iBqwLAAoErBKOIgasDN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDHyehNNb_eEw6-O-XMpFaJ4LE5kVF1RYT6Eif4dj4h5SDdGNwgqwLg3VkcIKsDYV3YWt1MgE
2026-03-09 04:35:56.182 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.47.128 waku f1e569abeeabb529488fecdc75d83976b92846ba9c3b84b29fdbeab766e89453
2026-03-09 04:35:56.217 DEBUG [src.node.docker_mananger] Container started with ID f1e569abeeab. Setting up logs at ./log/docker/node2_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:56.218 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 36917
2026-03-09 04:35:56.218 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:35:57.219 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36917/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:57.239 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:35:57.240 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:35:57.241 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36917/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:57.249 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.47.128/tcp/36918/p2p/16Uiu2HAm4yW1hp89GyhBSZzYBp28bajo1rWsY1JQrBNwzgL57cNk","/ip4/172.18.47.128/tcp/36919/ws/p2p/16Uiu2HAm4yW1hp89GyhBSZzYBp28bajo1rWsY1JQrBNwzgL57cNk"],"enrUri":"enr:-L24QIasBWA4puQtcM_o5gyNzwfVssj_Mvfk4415AhQEYmxkKpT8DcHb988D_VMaGfOaXyxliQhPW-y2wM3zAa0c0n4CgmlkgnY0gmlwhKwSL4CKbXVsdGlhZGRyc5YACASsEi-ABpA2AAoErBIvgAaQN90DgnJzhQADAQAAiXNlY3AyNTZrMaECjeB_miBVaoOsXQMNiRE4gheo6aidrSrkniWmiC9_vymDdGNwgpA2g3VkcIKQOIV3YWt1MgE"}'
2026-03-09 04:35:57.250 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:35:57.251 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:35:57.251 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:35:57.254 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:35:57.254 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.63.237
2026-03-09 04:35:57.254 DEBUG [src.node.docker_mananger] Generated ports ['61670', '61671', '61672', '61673', '61674']
2026-03-09 04:35:57.255 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:35:57.255 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:35:57.257 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:35:57.258 DEBUG [src.node.docker_mananger] docker run -i -t -p 61670:61670 -p 61671:61671 -p 61672:61672 -p 61673:61673 -p 61674:61674 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=61672 --rest-port=61670 --tcp-port=61671 --discv5-udp-port=61673 --rest-address=0.0.0.0 --nat=extip:172.18.63.237 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=bc5b6feed3fc2f7331185dbc514b219eaed0f18c81aabbdcb4af7dea2bb80103 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=61674 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QIasBWA4puQtcM_o5gyNzwfVssj_Mvfk4415AhQEYmxkKpT8DcHb988D_VMaGfOaXyxliQhPW-y2wM3zAa0c0n4CgmlkgnY0gmlwhKwSL4CKbXVsdGlhZGRyc5YACASsEi-ABpA2AAoErBIvgAaQN90DgnJzhQADAQAAiXNlY3AyNTZrMaECjeB_miBVaoOsXQMNiRE4gheo6aidrSrkniWmiC9_vymDdGNwgpA2g3VkcIKQOIV3YWt1MgE
2026-03-09 04:35:57.449 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.63.237 waku 36b9c42c5ec91729febb9593de65943cdfb4841a7566769410400d4efd88fceb
2026-03-09 04:35:57.485 DEBUG [src.node.docker_mananger] Container started with ID 36b9c42c5ec9. Setting up logs at ./log/docker/node3_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:57.486 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 61670
2026-03-09 04:35:57.486 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:35:58.488 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61670/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:58.502 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:35:58.502 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:35:58.503 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61670/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:58.509 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.63.237/tcp/61671/p2p/16Uiu2HAm6uu82LmbXtAVv6FiDuKGLUH4NDGtyaVhfhGE5Gd3zCZK","/ip4/172.18.63.237/tcp/61672/ws/p2p/16Uiu2HAm6uu82LmbXtAVv6FiDuKGLUH4NDGtyaVhfhGE5Gd3zCZK"],"enrUri":"enr:-L24QLrLmWYANZ7rak9msb2dLbpXKWPQMwGfulAEeI6yNeaSKDINfWjQi4EQNRHkMXLuTVLJ54iYw38-FOuWGhKT0iICgmlkgnY0gmlwhKwSP-2KbXVsdGlhZGRyc5YACASsEj_tBvDnAAoErBI_7Qbw6N0DgnJzhQADAQAAiXNlY3AyNTZrMaECqqutrl6Yy98jXoIfoHwH0cCL8pU9unX1buh7JJHEOiaDdGNwgvDng3VkcILw6YV3YWt1MgE"}'
2026-03-09 04:35:58.510 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:35:58.510 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:35:58.511 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:35:58.514 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:35:58.515 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.54.112
2026-03-09 04:35:58.515 DEBUG [src.node.docker_mananger] Generated ports ['45519', '45520', '45521', '45522', '45523']
2026-03-09 04:35:58.515 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:35:58.515 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:35:58.515 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:35:58.515 DEBUG [src.node.docker_mananger] docker run -i -t -p 45519:45519 -p 45520:45520 -p 45521:45521 -p 45522:45522 -p 45523:45523 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=45521 --rest-port=45519 --tcp-port=45520 --discv5-udp-port=45522 --rest-address=0.0.0.0 --nat=extip:172.18.54.112 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=bfcfd8cebff6c5abff1c0ddb59fcdc9649ee2adf0e14131a60a8440bf0ff00e4 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=45523 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLrLmWYANZ7rak9msb2dLbpXKWPQMwGfulAEeI6yNeaSKDINfWjQi4EQNRHkMXLuTVLJ54iYw38-FOuWGhKT0iICgmlkgnY0gmlwhKwSP-2KbXVsdGlhZGRyc5YACASsEj_tBvDnAAoErBI_7Qbw6N0DgnJzhQADAQAAiXNlY3AyNTZrMaECqqutrl6Yy98jXoIfoHwH0cCL8pU9unX1buh7JJHEOiaDdGNwgvDng3VkcILw6YV3YWt1MgE
2026-03-09 04:35:58.704 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.54.112 waku 373e94c3b2eca8da103100a83b4e8ae0d977e0df474a96a951acfbc45a6aac69
2026-03-09 04:35:58.742 DEBUG [src.node.docker_mananger] Container started with ID 373e94c3b2ec. Setting up logs at ./log/docker/node4_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:58.743 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 45519
2026-03-09 04:35:58.743 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:35:59.743 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45519/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:59.757 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:35:59.759 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:35:59.759 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45519/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:35:59.765 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.54.112/tcp/45520/p2p/16Uiu2HAkwZ42EWK5bGB6Lz19T7LuJ54f59W3Kms1dWVTyTyaoaiR","/ip4/172.18.54.112/tcp/45521/ws/p2p/16Uiu2HAkwZ42EWK5bGB6Lz19T7LuJ54f59W3Kms1dWVTyTyaoaiR"],"enrUri":"enr:-L24QC_IMuwBxTalJ_j-B_rdhmjjDEOLWjOXKCuOqvABJ304fvCc4y-VkBxsM61oouswOGpdeTbqS6hIOKTpePX4vTgCgmlkgnY0gmlwhKwSNnCKbXVsdGlhZGRyc5YACASsEjZwBrHQAAoErBI2cAax0d0DgnJzhQADAQAAiXNlY3AyNTZrMaECH5tc0SAifibr5lZWOtzlREKsKDBHarClgykQVc5ADMaDdGNwgrHQg3VkcIKx0oV3YWt1MgE"}'
2026-03-09 04:35:59.765 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:35:59.766 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:35:59.766 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:35:59.768 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:35:59.768 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.175.220
2026-03-09 04:35:59.769 DEBUG [src.node.docker_mananger] Generated ports ['27031', '27032', '27033', '27034', '27035']
2026-03-09 04:35:59.769 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:35:59.769 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:35:59.769 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:35:59.769 DEBUG [src.node.docker_mananger] docker run -i -t -p 27031:27031 -p 27032:27032 -p 27033:27033 -p 27034:27034 -p 27035:27035 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=27033 --rest-port=27031 --tcp-port=27032 --discv5-udp-port=27034 --rest-address=0.0.0.0 --nat=extip:172.18.175.220 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=eca0538c8eaf8744accaca8bc7733f6c0f6e00c1476ace50abcdeabedb0eb751 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=27035 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QC_IMuwBxTalJ_j-B_rdhmjjDEOLWjOXKCuOqvABJ304fvCc4y-VkBxsM61oouswOGpdeTbqS6hIOKTpePX4vTgCgmlkgnY0gmlwhKwSNnCKbXVsdGlhZGRyc5YACASsEjZwBrHQAAoErBI2cAax0d0DgnJzhQADAQAAiXNlY3AyNTZrMaECH5tc0SAifibr5lZWOtzlREKsKDBHarClgykQVc5ADMaDdGNwgrHQg3VkcIKx0oV3YWt1MgE
2026-03-09 04:35:59.965 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.175.220 waku 25997307b7910617a1e7338bfadd55f85d79d13eb5312462c5985cb4e0827007
2026-03-09 04:35:59.997 DEBUG [src.node.docker_mananger] Container started with ID 25997307b791. Setting up logs at ./log/docker/node5_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:35:59.998 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 27031
2026-03-09 04:35:59.998 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:36:01.000 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27031/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:01.016 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:36:01.017 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:36:01.017 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27031/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:01.025 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.175.220/tcp/27032/p2p/16Uiu2HAmVpe8umZoHXGTiaeooo3cxM4NC9Q2BjZ3FfUHut94jm9B","/ip4/172.18.175.220/tcp/27033/ws/p2p/16Uiu2HAmVpe8umZoHXGTiaeooo3cxM4NC9Q2BjZ3FfUHut94jm9B"],"enrUri":"enr:-L24QLO3KQyX4tDuXmUkJMC39c9vWR8FDlU2kEQcXMAe3CHwZmOlbW_nUmp5InLcxekIY2b9-hQMHu6lAWjPHdzTK1YCgmlkgnY0gmlwhKwSr9yKbXVsdGlhZGRyc5YACASsEq_cBmmYAAoErBKv3AZpmd0DgnJzhQADAQAAiXNlY3AyNTZrMaED_w-hZCT_-yVrTVgj3K6zz7L3o4X16WZ6mcDJYcm9WYqDdGNwgmmYg3VkcIJpmoV3YWt1MgE"}'
2026-03-09 04:36:01.025 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:36:01.026 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:36:01.028 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:36:01.029 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:36:01.029 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.168.240
2026-03-09 04:36:01.029 DEBUG [src.node.docker_mananger] Generated ports ['60723', '60724', '60725', '60726', '60727']
2026-03-09 04:36:01.029 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:36:01.030 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:36:01.030 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:36:01.030 DEBUG [src.node.docker_mananger] docker run -i -t -p 60723:60723 -p 60724:60724 -p 60725:60725 -p 60726:60726 -p 60727:60727 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=60725 --rest-port=60723 --tcp-port=60724 --discv5-udp-port=60726 --rest-address=0.0.0.0 --nat=extip:172.18.168.240 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=ce081ca2bcc04d7a1db72f4b5e2d4b29eb0cba74281b193fa3aee61a7abdfcf4 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=60727 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLO3KQyX4tDuXmUkJMC39c9vWR8FDlU2kEQcXMAe3CHwZmOlbW_nUmp5InLcxekIY2b9-hQMHu6lAWjPHdzTK1YCgmlkgnY0gmlwhKwSr9yKbXVsdGlhZGRyc5YACASsEq_cBmmYAAoErBKv3AZpmd0DgnJzhQADAQAAiXNlY3AyNTZrMaED_w-hZCT_-yVrTVgj3K6zz7L3o4X16WZ6mcDJYcm9WYqDdGNwgmmYg3VkcIJpmoV3YWt1MgE
2026-03-09 04:36:01.228 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.168.240 waku 7e929db2f4f3780503e5ca1fd7d82be998f8aab257a366fd4424cc412ab200d2
2026-03-09 04:36:01.267 DEBUG [src.node.docker_mananger] Container started with ID 7e929db2f4f3. Setting up logs at ./log/docker/node6_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:36:01.267 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 60723
2026-03-09 04:36:01.267 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:36:02.268 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60723/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:02.283 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:36:02.283 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:36:02.283 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60723/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:02.292 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.168.240/tcp/60724/p2p/16Uiu2HAm8Y9CDbjqJ1PHcLiecMEJB4rnjThYhvE4ETkq9SEzTpyh","/ip4/172.18.168.240/tcp/60725/ws/p2p/16Uiu2HAm8Y9CDbjqJ1PHcLiecMEJB4rnjThYhvE4ETkq9SEzTpyh"],"enrUri":"enr:-L24QHjLqKym_ZeiCSt8fZH2g86Rla4VdwjGCn-YhVpEGondSKXVI-k3f1hMiXss4aAIZ8BCR1Ktmulrxq0DTsAIFCgCgmlkgnY0gmlwhKwSqPCKbXVsdGlhZGRyc5YACASsEqjwBu00AAoErBKo8AbtNd0DgnJzhQADAQAAiXNlY3AyNTZrMaECwtAvpNfPf2z-4Ecr7_xOdMnHxbKII4IDfRMD7wJeHnSDdGNwgu00g3VkcILtNoV3YWt1MgE"}'
2026-03-09 04:36:02.292 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:36:02.293 DEBUG [src.node.waku_node] Starting Node...
2026-03-09 04:36:02.293 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-09 04:36:02.295 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-09 04:36:02.295 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.35.62
2026-03-09 04:36:02.295 DEBUG [src.node.docker_mananger] Generated ports ['58238', '58239', '58240', '58241', '58242']
2026-03-09 04:36:02.295 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-09 04:36:02.295 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-09 04:36:02.295 DEBUG [src.node.waku_node] Using volumes []
2026-03-09 04:36:02.296 DEBUG [src.node.docker_mananger] docker run -i -t -p 58238:58238 -p 58239:58239 -p 58240:58240 -p 58241:58241 -p 58242:58242 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=58240 --rest-port=58238 --tcp-port=58239 --discv5-udp-port=58241 --rest-address=0.0.0.0 --nat=extip:172.18.35.62 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=38e04916cc77fd66cdcea0cf3bc2edd986ca3e999a22ebddf3cabdedf2f4caca --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=58242 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHjLqKym_ZeiCSt8fZH2g86Rla4VdwjGCn-YhVpEGondSKXVI-k3f1hMiXss4aAIZ8BCR1Ktmulrxq0DTsAIFCgCgmlkgnY0gmlwhKwSqPCKbXVsdGlhZGRyc5YACASsEqjwBu00AAoErBKo8AbtNd0DgnJzhQADAQAAiXNlY3AyNTZrMaECwtAvpNfPf2z-4Ecr7_xOdMnHxbKII4IDfRMD7wJeHnSDdGNwgu00g3VkcILtNoV3YWt1MgE
2026-03-09 04:36:02.503 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.35.62 waku 1b754ea218de8c111ebcb152835dc933f21de1161ad49ed074cb86207a8ef69e
2026-03-09 04:36:02.539 DEBUG [src.node.docker_mananger] Container started with ID 1b754ea218de. Setting up logs at ./log/docker/node7_2026-03-09_04-35-54__a36d2a1c-be58-449d-96fa-66485dd982dc__wakuorg_nwaku:latest.log
2026-03-09 04:36:02.539 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 58238
2026-03-09 04:36:02.540 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-09 04:36:03.541 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58238/health" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.559 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-09 04:36:03.563 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-09 04:36:03.563 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58238/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.570 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.35.62/tcp/58239/p2p/16Uiu2HAmNnaHJdmCre9E5ihijPuaYkQ7jsnun3xpdbNfRA8VhFwZ","/ip4/172.18.35.62/tcp/58240/ws/p2p/16Uiu2HAmNnaHJdmCre9E5ihijPuaYkQ7jsnun3xpdbNfRA8VhFwZ"],"enrUri":"enr:-L24QO9JzEwRM_AQ2hk6JrT4Qn4gxnh9A_Swz6mZK2HBPQ8bUUtSGcQGb3yGPA17HpW_fx71wHwotNu8l_kY9AsTK3ICgmlkgnY0gmlwhKwSIz6KbXVsdGlhZGRyc5YACASsEiM-BuN_AAoErBIjPgbjgN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDloZNGPKzsArvDSC7sFNewIHWnVDR6ZaMgQSxVVVMM_SDdGNwguN_g3VkcILjgYV3YWt1MgE"}'
2026-03-09 04:36:03.570 INFO [src.node.waku_node] REST service is ready !!
2026-03-09 04:36:03.571 INFO [tests.e2e.test_network_conditions] Subscribing all nodes to relay topic
2026-03-09 04:36:03.571 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:44042/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.575 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.576 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:36917/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.583 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.584 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:61670/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.595 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.596 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:45519/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.607 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.607 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27031/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.614 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.614 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:60723/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.625 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.626 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58238/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-09 04:36:03.632 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-09 04:36:03.632 INFO [tests.e2e.test_network_conditions] Waiting for autoconnection
2026-03-09 04:36:03.635 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44042/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.638 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.47.128/tcp/55946/p2p/16Uiu2HAm4yW1hp89GyhBSZzYBp28bajo1rWsY1JQrBNwzgL57cNk","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-09 04:36:03.638 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:36917/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.640 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.63.237/tcp/56878/p2p/16Uiu2HAm6uu82LmbXtAVv6FiDuKGLUH4NDGtyaVhfhGE5Gd3zCZK","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.142.34/tcp/44043/p2p/16Uiu2HAmEkc3A1V4EtLN7azmYoBbys4pcGyxbE79oLvRxEyND2oM","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-09 04:36:03.640 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61670/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.643 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.47.128/tcp/36918/p2p/16Uiu2HAm4yW1hp89GyhBSZzYBp28bajo1rWsY1JQrBNwzgL57cNk","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.18.54.112/tcp/43148/p2p/16Uiu2HAkwZ42EWK5bGB6Lz19T7LuJ54f59W3Kms1dWVTyTyaoaiR","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-09 04:36:03.643 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:45519/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.645 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.175.220/tcp/54456/p2p/16Uiu2HAmVpe8umZoHXGTiaeooo3cxM4NC9Q2BjZ3FfUHut94jm9B","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.63.237/tcp/61671/p2p/16Uiu2HAm6uu82LmbXtAVv6FiDuKGLUH4NDGtyaVhfhGE5Gd3zCZK","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-09 04:36:03.646 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27031/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.648 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.168.240/tcp/39672/p2p/16Uiu2HAm8Y9CDbjqJ1PHcLiecMEJB4rnjThYhvE4ETkq9SEzTpyh","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.54.112/tcp/45520/p2p/16Uiu2HAkwZ42EWK5bGB6Lz19T7LuJ54f59W3Kms1dWVTyTyaoaiR","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-09 04:36:03.648 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:60723/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.651 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.175.220/tcp/27032/p2p/16Uiu2HAmVpe8umZoHXGTiaeooo3cxM4NC9Q2BjZ3FfUHut94jm9B","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.18.35.62/tcp/43970/p2p/16Uiu2HAmNnaHJdmCre9E5ihijPuaYkQ7jsnun3xpdbNfRA8VhFwZ","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-09 04:36:03.651 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58238/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:36:03.653 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.168.240/tcp/60724/p2p/16Uiu2HAm8Y9CDbjqJ1PHcLiecMEJB4rnjThYhvE4ETkq9SEzTpyh","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-09 04:36:03.653 DEBUG [src.libs.common] Sleeping for 60 seconds
2026-03-09 04:37:03.654 INFO [tests.e2e.test_network_conditions] Applying 3s latency to node3
2026-03-09 04:37:03.656 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4719', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-09 04:37:03.743 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4719', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'delay', '3000ms']
2026-03-09 04:37:03.757 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:61670/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-09 04:37:09.760 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-09 04:37:09.760 INFO [tests.e2e.test_network_conditions] Observed GET latency on node3: 6.00s
2026-03-09 04:37:09.762 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-09 04:37:09.763 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-09 04:37:09.763 DEBUG [src.node.waku_node] Stopping container with id 3cb9fe97dd6d
2026-03-09 04:37:10.402 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:10.405 DEBUG [src.node.waku_node] Stopping container with id f1e569abeeab
2026-03-09 04:37:10.997 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:10.999 DEBUG [src.node.waku_node] Stopping container with id 36b9c42c5ec9
2026-03-09 04:37:11.215 ERROR [src.node.docker_mananger] Max retries reached for container 3cb9fe97dd6d. Exiting log stream.
2026-03-09 04:37:11.594 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:11.594 DEBUG [src.node.waku_node] Stopping container with id 373e94c3b2ec
2026-03-09 04:37:11.814 ERROR [src.node.docker_mananger] Max retries reached for container f1e569abeeab. Exiting log stream.
2026-03-09 04:37:12.140 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:12.142 DEBUG [src.node.waku_node] Stopping container with id 25997307b791
2026-03-09 04:37:12.411 ERROR [src.node.docker_mananger] Max retries reached for container 36b9c42c5ec9. Exiting log stream.
2026-03-09 04:37:12.712 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:12.713 DEBUG [src.node.waku_node] Stopping container with id 7e929db2f4f3
2026-03-09 04:37:13.258 ERROR [src.node.docker_mananger] Max retries reached for container 373e94c3b2ec. Exiting log stream.
2026-03-09 04:37:13.267 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:13.269 DEBUG [src.node.waku_node] Stopping container with id 1b754ea218de
2026-03-09 04:37:13.525 ERROR [src.node.docker_mananger] Max retries reached for container 25997307b791. Exiting log stream.
2026-03-09 04:37:14.080 ERROR [src.node.docker_mananger] Max retries reached for container 7e929db2f4f3. Exiting log stream.
2026-03-09 04:37:23.868 DEBUG [src.node.waku_node] Container stopped.
2026-03-09 04:37:23.869 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-09 04:37:23.957 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:37:24.038 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:37:24.126 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:37:24.293 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:37:24.377 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:37:24.465 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-09 04:37:24.555 DEBUG [src.node.docker_mananger] No errors found in the waku logs.