226 lines
47 KiB
Plaintext

2026-03-26 04:46:52.098 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-26 04:46:52.101 DEBUG [tests.conftest] Running test: test_relay_7_nodes_3sec_latency with id: 2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300
2026-03-26 04:46:52.101 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-26 04:46:52.102 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-26 04:46:52.112 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.113 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.120 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.120 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.132 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.133 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.141 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.141 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.148 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.149 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.156 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.157 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.164 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.165 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node5_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.172 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.173 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node6_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.182 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-26 04:46:52.182 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node7_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.183 INFO [tests.e2e.test_network_conditions] Starting nodes with chain bootstrap
2026-03-26 04:46:52.183 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:52.183 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:52.185 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:52.185 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.52.155
2026-03-26 04:46:52.186 DEBUG [src.node.docker_mananger] Generated ports ['10518', '10519', '10520', '10521', '10522']
2026-03-26 04:46:52.186 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:52.186 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:52.186 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:52.187 DEBUG [src.node.docker_mananger] docker run -i -t -p 10518:10518 -p 10519:10519 -p 10520:10520 -p 10521:10521 -p 10522:10522 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=10520 --rest-port=10518 --tcp-port=10519 --discv5-udp-port=10521 --rest-address=0.0.0.0 --nat=extip:172.18.52.155 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=8be20ff7f5b2a41bf4e3409cceac5af0ddab6c5dadbd9a90bdd49f9ffe3fd9f9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=10522 --metrics-logging=true --relay=true
2026-03-26 04:46:52.292 ERROR [src.node.docker_mananger] Max retries reached for container 145edf731ccd. Exiting log stream.
2026-03-26 04:46:52.380 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.52.155 waku 3e78575c8b2b01f2b2c2a9f9246402c9eea2e907f75d4b23ccfcb1b2ff0ca5d5
2026-03-26 04:46:52.415 DEBUG [src.node.docker_mananger] Container started with ID 3e78575c8b2b. Setting up logs at ./log/docker/node1_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:52.415 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 10518
2026-03-26 04:46:52.415 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:53.021 ERROR [src.node.docker_mananger] Max retries reached for container 05bf98492332. Exiting log stream.
2026-03-26 04:46:53.416 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10518/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:53.419 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:53.419 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:53.419 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10518/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:53.421 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.52.155/tcp/10519/p2p/16Uiu2HAm1aLD8ykYg1zDC46TnfQpg3dErYSLs56GLTF65GqMck46","/ip4/172.18.52.155/tcp/10520/ws/p2p/16Uiu2HAm1aLD8ykYg1zDC46TnfQpg3dErYSLs56GLTF65GqMck46"],"enrUri":"enr:-L24QGMCUSpx7MDWodu8UzEFexOFATUrx9nrz1aAGxHb1VxQXVsTZp7o3zEMuqjyQ954FfW1qqJZgjV5I-Zp6z8in6sCgmlkgnY0gmlwhKwSNJuKbXVsdGlhZGRyc5YACASsEjSbBikXAAoErBI0mwYpGN0DgnJzhQADAQAAiXNlY3AyNTZrMaECW13-s1sO7y5aCMHmXghr60SBjkDaJl41ttQckmxAZTeDdGNwgikXg3VkcIIpGYV3YWt1MgE"}'
2026-03-26 04:46:53.422 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:53.422 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:53.422 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:53.423 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:53.424 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.23.6
2026-03-26 04:46:53.424 DEBUG [src.node.docker_mananger] Generated ports ['54571', '54572', '54573', '54574', '54575']
2026-03-26 04:46:53.424 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:53.424 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:53.424 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:53.424 DEBUG [src.node.docker_mananger] docker run -i -t -p 54571:54571 -p 54572:54572 -p 54573:54573 -p 54574:54574 -p 54575:54575 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=54573 --rest-port=54571 --tcp-port=54572 --discv5-udp-port=54574 --rest-address=0.0.0.0 --nat=extip:172.18.23.6 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=dec0b2d90638abeab1a3e6cbcd15f52d3cb410ef8b1c9baa5bbb31cc3e776b7a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=54575 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QGMCUSpx7MDWodu8UzEFexOFATUrx9nrz1aAGxHb1VxQXVsTZp7o3zEMuqjyQ954FfW1qqJZgjV5I-Zp6z8in6sCgmlkgnY0gmlwhKwSNJuKbXVsdGlhZGRyc5YACASsEjSbBikXAAoErBI0mwYpGN0DgnJzhQADAQAAiXNlY3AyNTZrMaECW13-s1sO7y5aCMHmXghr60SBjkDaJl41ttQckmxAZTeDdGNwgikXg3VkcIIpGYV3YWt1MgE
2026-03-26 04:46:53.626 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.23.6 waku 2d3ec33da23c69a40127cf4300b858a40fe6471cc65eaac78e96f7570cb413aa
2026-03-26 04:46:53.659 DEBUG [src.node.docker_mananger] Container started with ID 2d3ec33da23c. Setting up logs at ./log/docker/node2_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:53.660 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 54571
2026-03-26 04:46:53.660 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:54.661 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54571/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:54.680 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:54.683 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:54.683 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54571/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:54.688 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.23.6/tcp/54572/p2p/16Uiu2HAm3ZVGnbhKcK8HKmtciiWXGdDsry5zJQxHKvWDb9z1bE63","/ip4/172.18.23.6/tcp/54573/ws/p2p/16Uiu2HAm3ZVGnbhKcK8HKmtciiWXGdDsry5zJQxHKvWDb9z1bE63"],"enrUri":"enr:-L24QMR-wqWr4fDv0k5MS2TXqrnkSgI-Ib_A0MFC8svgXZ7GVFm6Li5210FRkQsKdkQLHXe4jlDULMvV5lkxQ8JFrsACgmlkgnY0gmlwhKwSFwaKbXVsdGlhZGRyc5YACASsEhcGBtUsAAoErBIXBgbVLd0DgnJzhQADAQAAiXNlY3AyNTZrMaECeN4JEeuS81hXX-YTGuQ01-S3ezJaLE0RKb4qYnU9buiDdGNwgtUsg3VkcILVLoV3YWt1MgE"}'
2026-03-26 04:46:54.689 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:54.690 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:54.690 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:54.693 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:54.693 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.206.173
2026-03-26 04:46:54.693 DEBUG [src.node.docker_mananger] Generated ports ['4299', '4300', '4301', '4302', '4303']
2026-03-26 04:46:54.694 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:54.694 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:54.694 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:54.694 DEBUG [src.node.docker_mananger] docker run -i -t -p 4299:4299 -p 4300:4300 -p 4301:4301 -p 4302:4302 -p 4303:4303 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4301 --rest-port=4299 --tcp-port=4300 --discv5-udp-port=4302 --rest-address=0.0.0.0 --nat=extip:172.18.206.173 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=2e9eef61a8ecef03e4ba21bda0b3beb96ec9905ef10af3e5d4a5255a4bffd7c3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4303 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMR-wqWr4fDv0k5MS2TXqrnkSgI-Ib_A0MFC8svgXZ7GVFm6Li5210FRkQsKdkQLHXe4jlDULMvV5lkxQ8JFrsACgmlkgnY0gmlwhKwSFwaKbXVsdGlhZGRyc5YACASsEhcGBtUsAAoErBIXBgbVLd0DgnJzhQADAQAAiXNlY3AyNTZrMaECeN4JEeuS81hXX-YTGuQ01-S3ezJaLE0RKb4qYnU9buiDdGNwgtUsg3VkcILVLoV3YWt1MgE
2026-03-26 04:46:54.887 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.206.173 waku 24ab2b7499aa66a1fa3b39a01ba0d4f7ce4a1ba216d004b7215cb6de1bdce6d1
2026-03-26 04:46:54.922 DEBUG [src.node.docker_mananger] Container started with ID 24ab2b7499aa. Setting up logs at ./log/docker/node3_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:54.922 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 4299
2026-03-26 04:46:54.923 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:55.924 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4299/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:55.937 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:55.940 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:55.941 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4299/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:55.946 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.206.173/tcp/4300/p2p/16Uiu2HAmFhdNQ2AWDVxRWeWPoqHnn1eyKFQdLhds15YYb7BK14Du","/ip4/172.18.206.173/tcp/4301/ws/p2p/16Uiu2HAmFhdNQ2AWDVxRWeWPoqHnn1eyKFQdLhds15YYb7BK14Du"],"enrUri":"enr:-L24QFxeMFkdlolpPo1xGY3tJCci88k_J6bJlTIcT51hyVFJLr9zx0c4iv62Kurg7Ecn9o-PFsvZ_jhbpL3ByVzcXgYCgmlkgnY0gmlwhKwSzq2KbXVsdGlhZGRyc5YACASsEs6tBhDMAAoErBLOrQYQzd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDLUAUfLMStmqTvPRHuCcQPdVNDnH4jXll3Zw-g2lp7TiDdGNwghDMg3VkcIIQzoV3YWt1MgE"}'
2026-03-26 04:46:55.947 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:55.947 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:55.948 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:55.950 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:55.951 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.215.11
2026-03-26 04:46:55.951 DEBUG [src.node.docker_mananger] Generated ports ['51119', '51120', '51121', '51122', '51123']
2026-03-26 04:46:55.951 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:55.951 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:55.952 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:55.953 DEBUG [src.node.docker_mananger] docker run -i -t -p 51119:51119 -p 51120:51120 -p 51121:51121 -p 51122:51122 -p 51123:51123 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=51121 --rest-port=51119 --tcp-port=51120 --discv5-udp-port=51122 --rest-address=0.0.0.0 --nat=extip:172.18.215.11 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=25fbc58db6dbf9022a4cceb394371fec4abfd7b1bedf70eee6dd2ac882cb5dde --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=51123 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QFxeMFkdlolpPo1xGY3tJCci88k_J6bJlTIcT51hyVFJLr9zx0c4iv62Kurg7Ecn9o-PFsvZ_jhbpL3ByVzcXgYCgmlkgnY0gmlwhKwSzq2KbXVsdGlhZGRyc5YACASsEs6tBhDMAAoErBLOrQYQzd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDLUAUfLMStmqTvPRHuCcQPdVNDnH4jXll3Zw-g2lp7TiDdGNwghDMg3VkcIIQzoV3YWt1MgE
2026-03-26 04:46:56.151 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.215.11 waku 39f052f9fd391ba5c9f35bd7366f228df28c079d96d1de4c6ecc9d68ec671364
2026-03-26 04:46:56.189 DEBUG [src.node.docker_mananger] Container started with ID 39f052f9fd39. Setting up logs at ./log/docker/node4_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:56.189 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 51119
2026-03-26 04:46:56.189 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:57.190 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51119/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:57.204 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:57.206 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:57.206 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51119/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:57.214 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.215.11/tcp/51120/p2p/16Uiu2HAmGUHKLQWgeKVQ1ppthzYRNppPCCyBdDsEdrTjNBZjcU8t","/ip4/172.18.215.11/tcp/51121/ws/p2p/16Uiu2HAmGUHKLQWgeKVQ1ppthzYRNppPCCyBdDsEdrTjNBZjcU8t"],"enrUri":"enr:-L24QGWFrtlQS39yOtYJykrJ_3QP6z5GYVHfdlcHr8TQw1IRDyJHI_CX9wqUZrA2xH8LCSEvlwWYt-CeP00RaveUh2MCgmlkgnY0gmlwhKwS1wuKbXVsdGlhZGRyc5YACASsEtcLBsewAAoErBLXCwbHsd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDOLCK7A4Ios9-vmfbbCD9KPC5XPnU9YlLrm5SX7RzDa2DdGNwgsewg3VkcILHsoV3YWt1MgE"}'
2026-03-26 04:46:57.215 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:57.215 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:57.215 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:57.217 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:57.217 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.122.169
2026-03-26 04:46:57.217 DEBUG [src.node.docker_mananger] Generated ports ['32611', '32612', '32613', '32614', '32615']
2026-03-26 04:46:57.217 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:57.218 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:57.218 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:57.218 DEBUG [src.node.docker_mananger] docker run -i -t -p 32611:32611 -p 32612:32612 -p 32613:32613 -p 32614:32614 -p 32615:32615 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=32613 --rest-port=32611 --tcp-port=32612 --discv5-udp-port=32614 --rest-address=0.0.0.0 --nat=extip:172.18.122.169 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e39e4613e03dab4f974bf7fce873b4afe8de4bdcee95a42e4e0dc42dec0babf2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=32615 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QGWFrtlQS39yOtYJykrJ_3QP6z5GYVHfdlcHr8TQw1IRDyJHI_CX9wqUZrA2xH8LCSEvlwWYt-CeP00RaveUh2MCgmlkgnY0gmlwhKwS1wuKbXVsdGlhZGRyc5YACASsEtcLBsewAAoErBLXCwbHsd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDOLCK7A4Ios9-vmfbbCD9KPC5XPnU9YlLrm5SX7RzDa2DdGNwgsewg3VkcILHsoV3YWt1MgE
2026-03-26 04:46:57.410 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.122.169 waku e23bbce9bbeaf54b3a411ff20636e81c59c1e26b79a31c9a9c330ce7746ddd62
2026-03-26 04:46:57.445 DEBUG [src.node.docker_mananger] Container started with ID e23bbce9bbea. Setting up logs at ./log/docker/node5_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:57.446 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 32611
2026-03-26 04:46:57.446 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:58.450 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32611/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:58.469 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:58.470 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:58.470 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32611/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:58.476 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.122.169/tcp/32612/p2p/16Uiu2HAmRatQfq8ETTcXSSm6vbc1vc2ADSF9RCNVS66xg3xZbqfJ","/ip4/172.18.122.169/tcp/32613/ws/p2p/16Uiu2HAmRatQfq8ETTcXSSm6vbc1vc2ADSF9RCNVS66xg3xZbqfJ"],"enrUri":"enr:-L24QMopexy1bRClOLoiqFGw2ZVI0R3G9_7NaCuuQhV3A2llCa6fFycEpqjLa4p0HKJXwMycxIzUahhBzHwRP-mHTlICgmlkgnY0gmlwhKwSeqmKbXVsdGlhZGRyc5YACASsEnqpBn9kAAoErBJ6qQZ_Zd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDwBrjc8nPB7RBod6PeqvMTJpbZz1ipuyYO-DmDgu02x2DdGNwgn9kg3VkcIJ_ZoV3YWt1MgE"}'
2026-03-26 04:46:58.477 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:58.477 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:58.478 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:58.481 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:58.481 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.162.133
2026-03-26 04:46:58.481 DEBUG [src.node.docker_mananger] Generated ports ['20509', '20510', '20511', '20512', '20513']
2026-03-26 04:46:58.482 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:58.482 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:58.483 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:58.483 DEBUG [src.node.docker_mananger] docker run -i -t -p 20509:20509 -p 20510:20510 -p 20511:20511 -p 20512:20512 -p 20513:20513 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=20511 --rest-port=20509 --tcp-port=20510 --discv5-udp-port=20512 --rest-address=0.0.0.0 --nat=extip:172.18.162.133 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fe3aaa7de579afcb9b27063e61cbd2ac3063a5bc3cab6ab6d1fffddca611e1ba --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=20513 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QMopexy1bRClOLoiqFGw2ZVI0R3G9_7NaCuuQhV3A2llCa6fFycEpqjLa4p0HKJXwMycxIzUahhBzHwRP-mHTlICgmlkgnY0gmlwhKwSeqmKbXVsdGlhZGRyc5YACASsEnqpBn9kAAoErBJ6qQZ_Zd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDwBrjc8nPB7RBod6PeqvMTJpbZz1ipuyYO-DmDgu02x2DdGNwgn9kg3VkcIJ_ZoV3YWt1MgE
2026-03-26 04:46:58.684 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.162.133 waku c68c7054267aeaa62debf0677bbdb81c1285248b9e1db9f1b020ef3d073b3c47
2026-03-26 04:46:58.723 DEBUG [src.node.docker_mananger] Container started with ID c68c7054267a. Setting up logs at ./log/docker/node6_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:58.723 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 20509
2026-03-26 04:46:58.724 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:46:59.724 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20509/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:59.741 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:46:59.746 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:46:59.748 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20509/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:46:59.752 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.162.133/tcp/20510/p2p/16Uiu2HAmP7tiRPsW7kSUGfhWwc92k1sCFVsvYJzRFXr8B2oTn5RM","/ip4/172.18.162.133/tcp/20511/ws/p2p/16Uiu2HAmP7tiRPsW7kSUGfhWwc92k1sCFVsvYJzRFXr8B2oTn5RM"],"enrUri":"enr:-L24QLDeodX6mhnH_BItPuSYLFwdUd6gUmowol3km6Fkv-l2IDqMIGDm60847a7EemaGjNP9ET4RL8_SFt0LB4ISyiECgmlkgnY0gmlwhKwSooWKbXVsdGlhZGRyc5YACASsEqKFBlAeAAoErBKihQZQH90DgnJzhQADAQAAiXNlY3AyNTZrMaEDm3ktyLU8SKokouaJecn9iQlFaTbxbL66OyBUWO_iQbyDdGNwglAeg3VkcIJQIIV3YWt1MgE"}'
2026-03-26 04:46:59.752 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:46:59.752 DEBUG [src.node.waku_node] Starting Node...
2026-03-26 04:46:59.753 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-26 04:46:59.754 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-26 04:46:59.754 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.244.1
2026-03-26 04:46:59.755 DEBUG [src.node.docker_mananger] Generated ports ['3086', '3087', '3088', '3089', '3090']
2026-03-26 04:46:59.755 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-26 04:46:59.755 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-26 04:46:59.755 DEBUG [src.node.waku_node] Using volumes []
2026-03-26 04:46:59.755 DEBUG [src.node.docker_mananger] docker run -i -t -p 3086:3086 -p 3087:3087 -p 3088:3088 -p 3089:3089 -p 3090:3090 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=3088 --rest-port=3086 --tcp-port=3087 --discv5-udp-port=3089 --rest-address=0.0.0.0 --nat=extip:172.18.244.1 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=dbb3c2d9e78ec4dce2f5ea78740a0ffe6513c59e573574e1ba2e8b931b6b0bac --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=3090 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLDeodX6mhnH_BItPuSYLFwdUd6gUmowol3km6Fkv-l2IDqMIGDm60847a7EemaGjNP9ET4RL8_SFt0LB4ISyiECgmlkgnY0gmlwhKwSooWKbXVsdGlhZGRyc5YACASsEqKFBlAeAAoErBKihQZQH90DgnJzhQADAQAAiXNlY3AyNTZrMaEDm3ktyLU8SKokouaJecn9iQlFaTbxbL66OyBUWO_iQbyDdGNwglAeg3VkcIJQIIV3YWt1MgE
2026-03-26 04:46:59.949 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.244.1 waku ad12e73297c1d40dec472380698f5e9864159301368989094c1425af0bdcb01a
2026-03-26 04:46:59.983 DEBUG [src.node.docker_mananger] Container started with ID ad12e73297c1. Setting up logs at ./log/docker/node7_2026-03-26_04-46-52__af5cf3b8-9593-4688-bb19-669d7b15a300__wakuorg_nwaku:latest.log
2026-03-26 04:46:59.984 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 3086
2026-03-26 04:46:59.984 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-26 04:47:00.985 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3086/health" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.006 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-26 04:47:01.009 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-26 04:47:01.010 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3086/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.016 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.244.1/tcp/3087/p2p/16Uiu2HAmHrUE4BCZVbPREHLUwLN2JsEFaBrGseujHb9uxckZLkAz","/ip4/172.18.244.1/tcp/3088/ws/p2p/16Uiu2HAmHrUE4BCZVbPREHLUwLN2JsEFaBrGseujHb9uxckZLkAz"],"enrUri":"enr:-L24QDS4I9_PHpWciiHeLXQqnClaMl7wRvdNKETCFwO8xSMSI7wY3wqLjzQz3R8sCh0hC3l5_RgnpGd0whR60K_rJucCgmlkgnY0gmlwhKwS9AGKbXVsdGlhZGRyc5YACASsEvQBBgwPAAoErBL0AQYMEN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDTTtXmnTcvaUGb4_ta9mIEjulTR0vuis24b1FBHM_dGeDdGNwggwPg3VkcIIMEYV3YWt1MgE"}'
2026-03-26 04:47:01.017 INFO [src.node.waku_node] REST service is ready !!
2026-03-26 04:47:01.017 INFO [tests.e2e.test_network_conditions] Subscribing all nodes to relay topic
2026-03-26 04:47:01.017 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:10518/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.022 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.022 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:54571/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.029 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.031 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4299/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.041 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.041 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:51119/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.050 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.051 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:32611/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.060 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.061 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20509/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.071 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.075 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:3086/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-26 04:47:01.081 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-26 04:47:01.082 INFO [tests.e2e.test_network_conditions] Waiting for autoconnection
2026-03-26 04:47:01.082 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:10518/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.087 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.23.6/tcp/56328/p2p/16Uiu2HAm3ZVGnbhKcK8HKmtciiWXGdDsry5zJQxHKvWDb9z1bE63","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-26 04:47:01.087 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:54571/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.089 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.52.155/tcp/10519/p2p/16Uiu2HAm1aLD8ykYg1zDC46TnfQpg3dErYSLs56GLTF65GqMck46","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.206.173/tcp/33640/p2p/16Uiu2HAmFhdNQ2AWDVxRWeWPoqHnn1eyKFQdLhds15YYb7BK14Du","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-26 04:47:01.090 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4299/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.092 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.215.11/tcp/56614/p2p/16Uiu2HAmGUHKLQWgeKVQ1ppthzYRNppPCCyBdDsEdrTjNBZjcU8t","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.23.6/tcp/54572/p2p/16Uiu2HAm3ZVGnbhKcK8HKmtciiWXGdDsry5zJQxHKvWDb9z1bE63","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-26 04:47:01.092 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:51119/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.095 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.206.173/tcp/4300/p2p/16Uiu2HAmFhdNQ2AWDVxRWeWPoqHnn1eyKFQdLhds15YYb7BK14Du","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.122.169/tcp/39846/p2p/16Uiu2HAmRatQfq8ETTcXSSm6vbc1vc2ADSF9RCNVS66xg3xZbqfJ","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-26 04:47:01.095 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:32611/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.097 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.215.11/tcp/51120/p2p/16Uiu2HAmGUHKLQWgeKVQ1ppthzYRNppPCCyBdDsEdrTjNBZjcU8t","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.162.133/tcp/45728/p2p/16Uiu2HAmP7tiRPsW7kSUGfhWwc92k1sCFVsvYJzRFXr8B2oTn5RM","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-26 04:47:01.097 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20509/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.100 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.244.1/tcp/43596/p2p/16Uiu2HAmHrUE4BCZVbPREHLUwLN2JsEFaBrGseujHb9uxckZLkAz","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.122.169/tcp/32612/p2p/16Uiu2HAmRatQfq8ETTcXSSm6vbc1vc2ADSF9RCNVS66xg3xZbqfJ","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-26 04:47:01.100 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:3086/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:47:01.102 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.162.133/tcp/20510/p2p/16Uiu2HAmP7tiRPsW7kSUGfhWwc92k1sCFVsvYJzRFXr8B2oTn5RM","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-26 04:47:01.102 DEBUG [src.libs.common] Sleeping for 60 seconds
2026-03-26 04:48:01.103 INFO [tests.e2e.test_network_conditions] Applying 3s latency to node3
2026-03-26 04:48:01.105 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '5477', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-26 04:48:01.156 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '5477', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'delay', '3000ms']
2026-03-26 04:48:01.170 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4299/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-26 04:48:07.173 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-26 04:48:07.174 INFO [tests.e2e.test_network_conditions] Observed GET latency on node3: 6.00s
2026-03-26 04:48:07.175 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-26 04:48:07.176 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-26 04:48:07.176 DEBUG [src.node.waku_node] Stopping container with id 3e78575c8b2b
2026-03-26 04:48:07.787 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:07.788 DEBUG [src.node.waku_node] Stopping container with id 2d3ec33da23c
2026-03-26 04:48:08.403 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:08.403 DEBUG [src.node.waku_node] Stopping container with id 24ab2b7499aa
2026-03-26 04:48:08.605 ERROR [src.node.docker_mananger] Max retries reached for container 3e78575c8b2b. Exiting log stream.
2026-03-26 04:48:08.965 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:08.967 DEBUG [src.node.waku_node] Stopping container with id 39f052f9fd39
2026-03-26 04:48:09.219 ERROR [src.node.docker_mananger] Max retries reached for container 2d3ec33da23c. Exiting log stream.
2026-03-26 04:48:09.542 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:09.545 DEBUG [src.node.waku_node] Stopping container with id e23bbce9bbea
2026-03-26 04:48:10.105 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:10.106 DEBUG [src.node.waku_node] Stopping container with id c68c7054267a
2026-03-26 04:48:10.112 ERROR [src.node.docker_mananger] Max retries reached for container 24ab2b7499aa. Exiting log stream.
2026-03-26 04:48:10.356 ERROR [src.node.docker_mananger] Max retries reached for container 39f052f9fd39. Exiting log stream.
2026-03-26 04:48:10.716 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:10.719 DEBUG [src.node.waku_node] Stopping container with id ad12e73297c1
2026-03-26 04:48:10.926 ERROR [src.node.docker_mananger] Max retries reached for container e23bbce9bbea. Exiting log stream.
2026-03-26 04:48:11.532 ERROR [src.node.docker_mananger] Max retries reached for container c68c7054267a. Exiting log stream.
2026-03-26 04:48:21.256 DEBUG [src.node.waku_node] Container stopped.
2026-03-26 04:48:21.258 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-26 04:48:21.340 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:21.420 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:21.581 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:21.664 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:21.746 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:21.830 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-26 04:48:21.911 DEBUG [src.node.docker_mananger] No errors found in the waku logs.