494 lines
98 KiB
Plaintext

2026-03-27 04:48:13.980 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-27 04:48:13.980 DEBUG [tests.conftest] Running test: test_relay_packet_loss_correlated_vs_uncorrelated with id: 2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e
2026-03-27 04:48:13.980 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-27 04:48:13.981 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-27 04:48:13.988 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:48:13.989 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:13.996 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:48:13.996 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:14.002 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:48:14.003 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:14.009 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:48:14.009 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:14.009 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:48:14.009 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:48:14.010 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:48:14.010 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.203.13
2026-03-27 04:48:14.011 DEBUG [src.node.docker_mananger] Generated ports ['48392', '48393', '48394', '48395', '48396']
2026-03-27 04:48:14.011 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:48:14.011 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:48:14.011 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:48:14.011 DEBUG [src.node.docker_mananger] docker run -i -t -p 48392:48392 -p 48393:48393 -p 48394:48394 -p 48395:48395 -p 48396:48396 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=48394 --rest-port=48392 --tcp-port=48393 --discv5-udp-port=48395 --rest-address=0.0.0.0 --nat=extip:172.18.203.13 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=4dcdbce7ebd7b1a2ccda56f3e36bd982b1d3ba50bdbead22f6b4c0de2bbcf14a --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=48396 --metrics-logging=true --relay=true
2026-03-27 04:48:14.192 ERROR [src.node.docker_mananger] Max retries reached for container e3faa6ca7947. Exiting log stream.
2026-03-27 04:48:14.201 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.203.13 waku 680196f03579ca7a4024710bf20bf9c8cb7599ceec0e3fd60c097856c09db970
2026-03-27 04:48:14.236 DEBUG [src.node.docker_mananger] Container started with ID 680196f03579. Setting up logs at ./log/docker/node1_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:14.236 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 48392
2026-03-27 04:48:14.237 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:48:14.760 ERROR [src.node.docker_mananger] Max retries reached for container 95049409f488. Exiting log stream.
2026-03-27 04:48:15.237 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48392/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:15.240 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:48:15.240 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:48:15.240 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48392/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:15.243 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.203.13/tcp/48393/p2p/16Uiu2HAm85sXJbBpriYA8djzgMoxPMoq5EYGYHxcfVPBdEzQueG4","/ip4/172.18.203.13/tcp/48394/ws/p2p/16Uiu2HAm85sXJbBpriYA8djzgMoxPMoq5EYGYHxcfVPBdEzQueG4"],"enrUri":"enr:-L24QIfhZ_SlPTghbbeTJo2iOqykkUsLEOxxJKRIvkXMBF36G_-EPORZgxq4rYWbsJ3UGSAdPDkvUvFhVb1MA-jpgUUCgmlkgnY0gmlwhKwSyw2KbXVsdGlhZGRyc5YACASsEssNBr0JAAoErBLLDQa9Ct0DgnJzhQADAQAAiXNlY3AyNTZrMaECvBVdq1esSJeaSp8QnnSAP3ph39vzx4j3HG1ZDe7M2A2DdGNwgr0Jg3VkcIK9C4V3YWt1MgE"}'
2026-03-27 04:48:15.243 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:48:15.243 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:48:15.243 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:48:15.245 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:48:15.245 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.91.26
2026-03-27 04:48:15.245 DEBUG [src.node.docker_mananger] Generated ports ['64483', '64484', '64485', '64486', '64487']
2026-03-27 04:48:15.245 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:48:15.246 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:48:15.246 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:48:15.246 DEBUG [src.node.docker_mananger] docker run -i -t -p 64483:64483 -p 64484:64484 -p 64485:64485 -p 64486:64486 -p 64487:64487 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=64485 --rest-port=64483 --tcp-port=64484 --discv5-udp-port=64486 --rest-address=0.0.0.0 --nat=extip:172.18.91.26 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=0c1d2ea48df4d794dbc7613cb0ecf1addf426fc65544e2ad4afce62f44bd3ba6 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=64487 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QIfhZ_SlPTghbbeTJo2iOqykkUsLEOxxJKRIvkXMBF36G_-EPORZgxq4rYWbsJ3UGSAdPDkvUvFhVb1MA-jpgUUCgmlkgnY0gmlwhKwSyw2KbXVsdGlhZGRyc5YACASsEssNBr0JAAoErBLLDQa9Ct0DgnJzhQADAQAAiXNlY3AyNTZrMaECvBVdq1esSJeaSp8QnnSAP3ph39vzx4j3HG1ZDe7M2A2DdGNwgr0Jg3VkcIK9C4V3YWt1MgE
2026-03-27 04:48:15.438 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.91.26 waku 8670de0f13e4485e68906ff6f7d7c171bdade092f7e245f644225ae1ecf14bce
2026-03-27 04:48:15.474 DEBUG [src.node.docker_mananger] Container started with ID 8670de0f13e4. Setting up logs at ./log/docker/node2_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:15.474 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 64483
2026-03-27 04:48:15.474 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:48:16.474 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64483/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:16.497 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:48:16.501 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:48:16.502 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64483/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:16.506 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.91.26/tcp/64484/p2p/16Uiu2HAm9fZGiWy5YiWM5LEixdXQ1tkKeBMrub4ugbt3Q2mqmwvo","/ip4/172.18.91.26/tcp/64485/ws/p2p/16Uiu2HAm9fZGiWy5YiWM5LEixdXQ1tkKeBMrub4ugbt3Q2mqmwvo"],"enrUri":"enr:-L24QHfpHJ6VtlhrWeX6Dbdg-bqvLQC9Ucj3NcbJd0ePz5cELUBXigETqSPJssui7Q5g2QSnsLi7UbINHcgVd9j92IkCgmlkgnY0gmlwhKwSWxqKbXVsdGlhZGRyc5YACASsElsaBvvkAAoErBJbGgb75d0DgnJzhQADAQAAiXNlY3AyNTZrMaEC05IpzQhSH7XLneKd36bflCFCvxa9R-lRL9uITb1BAuiDdGNwgvvkg3VkcIL75oV3YWt1MgE"}'
2026-03-27 04:48:16.506 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:48:16.507 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:48:16.507 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:48:16.508 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:48:16.508 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.66.34
2026-03-27 04:48:16.508 DEBUG [src.node.docker_mananger] Generated ports ['20610', '20611', '20612', '20613', '20614']
2026-03-27 04:48:16.509 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:48:16.509 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:48:16.509 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:48:16.509 DEBUG [src.node.docker_mananger] docker run -i -t -p 20610:20610 -p 20611:20611 -p 20612:20612 -p 20613:20613 -p 20614:20614 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=20612 --rest-port=20610 --tcp-port=20611 --discv5-udp-port=20613 --rest-address=0.0.0.0 --nat=extip:172.18.66.34 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=5bfaf9508a84a3dcc4ef3dbbd20c79a0cf93da894e2a8bb6cea5d8f2eeb5ecb9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=20614 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHfpHJ6VtlhrWeX6Dbdg-bqvLQC9Ucj3NcbJd0ePz5cELUBXigETqSPJssui7Q5g2QSnsLi7UbINHcgVd9j92IkCgmlkgnY0gmlwhKwSWxqKbXVsdGlhZGRyc5YACASsElsaBvvkAAoErBJbGgb75d0DgnJzhQADAQAAiXNlY3AyNTZrMaEC05IpzQhSH7XLneKd36bflCFCvxa9R-lRL9uITb1BAuiDdGNwgvvkg3VkcIL75oV3YWt1MgE
2026-03-27 04:48:16.698 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.66.34 waku 6ce5b3be5884142f16a85639dab55577c71a63c998e44b3a199a648fafee21cd
2026-03-27 04:48:16.737 DEBUG [src.node.docker_mananger] Container started with ID 6ce5b3be5884. Setting up logs at ./log/docker/node3_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:16.738 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 20610
2026-03-27 04:48:16.738 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:48:17.738 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20610/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:17.749 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:48:17.752 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:48:17.752 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20610/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:17.758 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.66.34/tcp/20611/p2p/16Uiu2HAmSU1MvfoGy9ZDKdH9xToDLN1cu2b8uTc7yeT2cG1JWKF1","/ip4/172.18.66.34/tcp/20612/ws/p2p/16Uiu2HAmSU1MvfoGy9ZDKdH9xToDLN1cu2b8uTc7yeT2cG1JWKF1"],"enrUri":"enr:-L24QEmgXF1v4zwIbbTMfrXMay3i5r9was6EYJzFK7LIM8gML2pRHm-bnx4QsbSfOmM7uBbLwhKtzSNyTVM7dHbuo-0CgmlkgnY0gmlwhKwSQiKKbXVsdGlhZGRyc5YACASsEkIiBlCDAAoErBJCIgZQhN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDzTNfBW8ICU0tOt9YZ7P0MahLGgonxFPWUi2lUCIzioyDdGNwglCDg3VkcIJQhYV3YWt1MgE"}'
2026-03-27 04:48:17.758 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:48:17.758 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:48:17.759 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:48:17.761 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:48:17.761 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.12.54
2026-03-27 04:48:17.761 DEBUG [src.node.docker_mananger] Generated ports ['50739', '50740', '50741', '50742', '50743']
2026-03-27 04:48:17.762 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:48:17.762 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:48:17.762 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:48:17.762 DEBUG [src.node.docker_mananger] docker run -i -t -p 50739:50739 -p 50740:50740 -p 50741:50741 -p 50742:50742 -p 50743:50743 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=50741 --rest-port=50739 --tcp-port=50740 --discv5-udp-port=50742 --rest-address=0.0.0.0 --nat=extip:172.18.12.54 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f930d7ae01d7feafbdfee7d3ec707cf3ba2d18db76aac28f81e0818aa6350bb9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=50743 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QEmgXF1v4zwIbbTMfrXMay3i5r9was6EYJzFK7LIM8gML2pRHm-bnx4QsbSfOmM7uBbLwhKtzSNyTVM7dHbuo-0CgmlkgnY0gmlwhKwSQiKKbXVsdGlhZGRyc5YACASsEkIiBlCDAAoErBJCIgZQhN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDzTNfBW8ICU0tOt9YZ7P0MahLGgonxFPWUi2lUCIzioyDdGNwglCDg3VkcIJQhYV3YWt1MgE
2026-03-27 04:48:17.957 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.12.54 waku 245411686a1ed6975b4ea68da025afca71d99ecf3f360d71881b9c6247df17a2
2026-03-27 04:48:17.997 DEBUG [src.node.docker_mananger] Container started with ID 245411686a1e. Setting up logs at ./log/docker/node4_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:48:17.997 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 50739
2026-03-27 04:48:17.997 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:48:18.998 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50739/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:19.013 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:48:19.015 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:48:19.016 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50739/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:19.021 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.12.54/tcp/50740/p2p/16Uiu2HAmRQymb9M2E1YDrdCY74Ddgh8JUjYjaKBT8rmbGRQj12XB","/ip4/172.18.12.54/tcp/50741/ws/p2p/16Uiu2HAmRQymb9M2E1YDrdCY74Ddgh8JUjYjaKBT8rmbGRQj12XB"],"enrUri":"enr:-L24QNeZXy1saKkWwFM5MYgm-8qkJN18Kg0QCMeJcM7aI1NOElY4EF5QSwbz4lrvgNk3GOfpM4_HXEuw_f-6cVBbJU8CgmlkgnY0gmlwhKwSDDaKbXVsdGlhZGRyc5YACASsEgw2BsY0AAoErBIMNgbGNd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDvZEkSHMcEM30l_NucNfeExgy4L9WXCuHeaXGAM4Nk3qDdGNwgsY0g3VkcILGNoV3YWt1MgE"}'
2026-03-27 04:48:19.022 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:48:19.022 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:48:19.030 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:19.030 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:64483/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:48:19.036 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:19.037 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:20610/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:48:19.045 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:19.046 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:50739/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:48:19.054 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:19.054 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:48392/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:19.058 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.91.26/tcp/40818/p2p/16Uiu2HAm9fZGiWy5YiWM5LEixdXQ1tkKeBMrub4ugbt3Q2mqmwvo","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:48:19.059 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:64483/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:19.061 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.66.34/tcp/51422/p2p/16Uiu2HAmSU1MvfoGy9ZDKdH9xToDLN1cu2b8uTc7yeT2cG1JWKF1","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.203.13/tcp/48393/p2p/16Uiu2HAm85sXJbBpriYA8djzgMoxPMoq5EYGYHxcfVPBdEzQueG4","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-27 04:48:19.061 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:20610/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:19.064 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.91.26/tcp/64484/p2p/16Uiu2HAm9fZGiWy5YiWM5LEixdXQ1tkKeBMrub4ugbt3Q2mqmwvo","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.12.54/tcp/36006/p2p/16Uiu2HAmRQymb9M2E1YDrdCY74Ddgh8JUjYjaKBT8rmbGRQj12XB","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:48:19.064 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50739/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:19.066 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.66.34/tcp/20611/p2p/16Uiu2HAmSU1MvfoGy9ZDKdH9xToDLN1cu2b8uTc7yeT2cG1JWKF1","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-27 04:48:19.066 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-27 04:48:39.068 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8254', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:48:39.080 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8254', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%']
2026-03-27 04:48:39.091 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8254', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-27 04:48:39.100 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8002: root refcnt 5 limit 1000 loss 50%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-27 04:48:39.100 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50739/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:48:39.103 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-27 04:48:39.103 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:48:39.110 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:39.112 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:48:40.360 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:40.360 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:48:41.840 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:41.840 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:48:42.463 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:42.464 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:48:43.488 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:48:43.488 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-27 04:49:13.489 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50739/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:13.491 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774586919103867617,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774586919111914379,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774586920360488364,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774586921840461150,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774586922464340818,"ephemeral":false,"proof":""}]'
2026-03-27 04:49:13.493 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8254', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:49:13.504 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8254', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:49:13.516 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8254', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%', '75.0%']
2026-03-27 04:49:13.526 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:50739/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:13.528 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-27 04:49:13.529 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:49:13.536 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:49:13.538 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:48392/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:49:33.818 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-27 04:49:33.819 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-27 04:49:33.819 DEBUG [src.node.waku_node] Stopping container with id 680196f03579
2026-03-27 04:49:34.450 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:49:34.452 DEBUG [src.node.waku_node] Stopping container with id 8670de0f13e4
2026-03-27 04:49:35.029 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:49:35.032 DEBUG [src.node.waku_node] Stopping container with id 6ce5b3be5884
2026-03-27 04:49:35.462 ERROR [src.node.docker_mananger] Max retries reached for container 680196f03579. Exiting log stream.
2026-03-27 04:49:35.618 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:49:35.619 DEBUG [src.node.waku_node] Stopping container with id 245411686a1e
2026-03-27 04:49:35.843 ERROR [src.node.docker_mananger] Max retries reached for container 8670de0f13e4. Exiting log stream.
2026-03-27 04:49:36.167 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:49:36.170 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-27 04:49:36.280 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:49:36.339 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:49:36.401 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:49:36.462 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:49:36.464 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-27 04:49:36.464 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-27 04:49:36.465 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.466 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.469 DEBUG [src.libs.common] Attaching file ./log/docker/node4_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.471 DEBUG [src.libs.common] Attaching file ./log/docker/node3_2026-03-27_04-48-13__c8aea52f-9d9d-4ae2-99f8-5ecc22231a2e__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.475 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-27 04:49:36.475 DEBUG [tests.conftest] Running test: test_relay_packet_loss_correlated_vs_uncorrelated with id: 2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058
2026-03-27 04:49:36.475 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-27 04:49:36.476 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-27 04:49:36.482 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:49:36.483 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.489 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:49:36.489 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.496 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:49:36.496 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.502 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:49:36.502 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.503 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:49:36.503 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:49:36.504 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:49:36.504 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.69.208
2026-03-27 04:49:36.504 DEBUG [src.node.docker_mananger] Generated ports ['2104', '2105', '2106', '2107', '2108']
2026-03-27 04:49:36.505 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:49:36.505 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:49:36.505 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:49:36.505 DEBUG [src.node.docker_mananger] docker run -i -t -p 2104:2104 -p 2105:2105 -p 2106:2106 -p 2107:2107 -p 2108:2108 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2106 --rest-port=2104 --tcp-port=2105 --discv5-udp-port=2107 --rest-address=0.0.0.0 --nat=extip:172.18.69.208 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fdfdabbeeeeb7f5d96fb3fa9ad688cdde3161a0c8a7dfe39d01f314be0a1c0da --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2108 --metrics-logging=true --relay=true
2026-03-27 04:49:36.605 ERROR [src.node.docker_mananger] Max retries reached for container 6ce5b3be5884. Exiting log stream.
2026-03-27 04:49:36.720 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.69.208 waku 4e93a6e6806ea57d083a1f76718335bc266e87b01429403ff47f5e1b95767891
2026-03-27 04:49:36.758 DEBUG [src.node.docker_mananger] Container started with ID 4e93a6e6806e. Setting up logs at ./log/docker/node1_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:36.759 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2104
2026-03-27 04:49:36.759 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:49:37.072 ERROR [src.node.docker_mananger] Max retries reached for container 245411686a1e. Exiting log stream.
2026-03-27 04:49:37.761 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2104/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:37.764 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:49:37.764 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:49:37.764 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2104/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:37.766 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.69.208/tcp/2105/p2p/16Uiu2HAmVQrzurz6mVvbmBJV2ZnQajBV3fu9HWUAoekxoNrsDBUU","/ip4/172.18.69.208/tcp/2106/ws/p2p/16Uiu2HAmVQrzurz6mVvbmBJV2ZnQajBV3fu9HWUAoekxoNrsDBUU"],"enrUri":"enr:-L24QPaVm6lfWrb_nsOJfcamuJ5eEJTq2nS30fmwj57VjSHvbjJ6J0BswTY4flM-eWDb2v3OdlvnhwQSB-aU1La8Ri4CgmlkgnY0gmlwhKwSRdCKbXVsdGlhZGRyc5YACASsEkXQBgg5AAoErBJF0AYIOt0DgnJzhQADAQAAiXNlY3AyNTZrMaED-Pg79t6dEPtzXtRJMi-TvHo_o8kF7VzwknaqDrnEw4GDdGNwggg5g3VkcIIIO4V3YWt1MgE"}'
2026-03-27 04:49:37.767 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:49:37.767 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:49:37.767 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:49:37.768 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:49:37.769 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.140.157
2026-03-27 04:49:37.769 DEBUG [src.node.docker_mananger] Generated ports ['21222', '21223', '21224', '21225', '21226']
2026-03-27 04:49:37.769 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:49:37.769 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:49:37.769 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:49:37.769 DEBUG [src.node.docker_mananger] docker run -i -t -p 21222:21222 -p 21223:21223 -p 21224:21224 -p 21225:21225 -p 21226:21226 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=21224 --rest-port=21222 --tcp-port=21223 --discv5-udp-port=21225 --rest-address=0.0.0.0 --nat=extip:172.18.140.157 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=b20147cbb76e55c4d1504d86fdf22c7d18253c6c710f0aea25fbc57f99b65dea --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=21226 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPaVm6lfWrb_nsOJfcamuJ5eEJTq2nS30fmwj57VjSHvbjJ6J0BswTY4flM-eWDb2v3OdlvnhwQSB-aU1La8Ri4CgmlkgnY0gmlwhKwSRdCKbXVsdGlhZGRyc5YACASsEkXQBgg5AAoErBJF0AYIOt0DgnJzhQADAQAAiXNlY3AyNTZrMaED-Pg79t6dEPtzXtRJMi-TvHo_o8kF7VzwknaqDrnEw4GDdGNwggg5g3VkcIIIO4V3YWt1MgE
2026-03-27 04:49:37.960 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.140.157 waku 3efc55b407f4a76dfd38d7b811582cffd0e0765f7262a6486a794e5d0a09b035
2026-03-27 04:49:37.995 DEBUG [src.node.docker_mananger] Container started with ID 3efc55b407f4. Setting up logs at ./log/docker/node2_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:37.996 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 21222
2026-03-27 04:49:37.996 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:49:38.996 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21222/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:39.015 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:49:39.016 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:49:39.016 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21222/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:39.024 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.140.157/tcp/21223/p2p/16Uiu2HAkzZ4AAgCw8NvKavsiSXQr18gKaPVfQTS3JAb42J6ahrDX","/ip4/172.18.140.157/tcp/21224/ws/p2p/16Uiu2HAkzZ4AAgCw8NvKavsiSXQr18gKaPVfQTS3JAb42J6ahrDX"],"enrUri":"enr:-L24QHtB7QgBI8FOW-YwpGTlWkTXIg8XdQYbWcyNQYy9JauTSd0KaCJPmzpCs_bb-Uk88PHScE1GHyZGguaLAHDjmU0CgmlkgnY0gmlwhKwSjJ2KbXVsdGlhZGRyc5YACASsEoydBlLnAAoErBKMnQZS6N0DgnJzhQADAQAAiXNlY3AyNTZrMaECTC6T-Oy5nD9QRIkL0uOA7p2te0J57NOyZ3IBPzwkb-qDdGNwglLng3VkcIJS6YV3YWt1MgE"}'
2026-03-27 04:49:39.024 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:49:39.025 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:49:39.025 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:49:39.028 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:49:39.028 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.255.210
2026-03-27 04:49:39.028 DEBUG [src.node.docker_mananger] Generated ports ['8460', '8461', '8462', '8463', '8464']
2026-03-27 04:49:39.029 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:49:39.029 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:49:39.029 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:49:39.029 DEBUG [src.node.docker_mananger] docker run -i -t -p 8460:8460 -p 8461:8461 -p 8462:8462 -p 8463:8463 -p 8464:8464 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=8462 --rest-port=8460 --tcp-port=8461 --discv5-udp-port=8463 --rest-address=0.0.0.0 --nat=extip:172.18.255.210 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=b1bab8b51ab32cddc24f3e28f3c2fd783d0a9eeb8c3bceee71b5306bd81a0dee --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=8464 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QHtB7QgBI8FOW-YwpGTlWkTXIg8XdQYbWcyNQYy9JauTSd0KaCJPmzpCs_bb-Uk88PHScE1GHyZGguaLAHDjmU0CgmlkgnY0gmlwhKwSjJ2KbXVsdGlhZGRyc5YACASsEoydBlLnAAoErBKMnQZS6N0DgnJzhQADAQAAiXNlY3AyNTZrMaECTC6T-Oy5nD9QRIkL0uOA7p2te0J57NOyZ3IBPzwkb-qDdGNwglLng3VkcIJS6YV3YWt1MgE
2026-03-27 04:49:39.231 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.255.210 waku 41af71e6376c6288dc424c46bae37d005133787d8422f0e804fe0b7191ccd99f
2026-03-27 04:49:39.268 DEBUG [src.node.docker_mananger] Container started with ID 41af71e6376c. Setting up logs at ./log/docker/node3_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:39.268 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 8460
2026-03-27 04:49:39.268 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:49:40.270 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:8460/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:40.287 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:49:40.289 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:49:40.290 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:8460/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:40.295 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.255.210/tcp/8461/p2p/16Uiu2HAmHDZiG9cxy3MB7kdy7zVdrqfy9HoCPrPuLBj1CBo1CJ9o","/ip4/172.18.255.210/tcp/8462/ws/p2p/16Uiu2HAmHDZiG9cxy3MB7kdy7zVdrqfy9HoCPrPuLBj1CBo1CJ9o"],"enrUri":"enr:-L24QK5wUUdkALfyi5bq248-ggVsxtKFSd6OSIGHYkebgFC4MSrhGtuQZCdFJfSIrEUOb4UdkpZcYFYxZZF9b-ST9VYCgmlkgnY0gmlwhKwS_9KKbXVsdGlhZGRyc5YACASsEv_SBiENAAoErBL_0gYhDt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDQ8cOP_7-K38PCzjvaQtVVbj3iVO-YNXWHCquBbkNudqDdGNwgiENg3VkcIIhD4V3YWt1MgE"}'
2026-03-27 04:49:40.295 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:49:40.296 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:49:40.298 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:49:40.304 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:49:40.304 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.76.53
2026-03-27 04:49:40.305 DEBUG [src.node.docker_mananger] Generated ports ['58447', '58448', '58449', '58450', '58451']
2026-03-27 04:49:40.305 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:49:40.305 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:49:40.305 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:49:40.305 DEBUG [src.node.docker_mananger] docker run -i -t -p 58447:58447 -p 58448:58448 -p 58449:58449 -p 58450:58450 -p 58451:58451 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=58449 --rest-port=58447 --tcp-port=58448 --discv5-udp-port=58450 --rest-address=0.0.0.0 --nat=extip:172.18.76.53 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e39ad42d58c5ab9ad7d6bf07880ade080b5c23fb6f4bcdaddbb88cdae0cef9fc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=58451 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QK5wUUdkALfyi5bq248-ggVsxtKFSd6OSIGHYkebgFC4MSrhGtuQZCdFJfSIrEUOb4UdkpZcYFYxZZF9b-ST9VYCgmlkgnY0gmlwhKwS_9KKbXVsdGlhZGRyc5YACASsEv_SBiENAAoErBL_0gYhDt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDQ8cOP_7-K38PCzjvaQtVVbj3iVO-YNXWHCquBbkNudqDdGNwgiENg3VkcIIhD4V3YWt1MgE
2026-03-27 04:49:40.531 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.76.53 waku 16b09238887fb59b8bc672395eaae519adea329a01fdc9ce7e8402fbaf28b1e9
2026-03-27 04:49:40.570 DEBUG [src.node.docker_mananger] Container started with ID 16b09238887f. Setting up logs at ./log/docker/node4_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:49:40.571 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 58447
2026-03-27 04:49:40.572 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:49:41.572 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58447/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:41.583 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:49:41.585 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:49:41.586 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58447/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:41.594 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.76.53/tcp/58448/p2p/16Uiu2HAmKyPDwDucYdkzSBsMXL7ZBXCn7kHiwwRyx94NuYuoev1h","/ip4/172.18.76.53/tcp/58449/ws/p2p/16Uiu2HAmKyPDwDucYdkzSBsMXL7ZBXCn7kHiwwRyx94NuYuoev1h"],"enrUri":"enr:-L24QF0zPVLiSBQB4fxpAmPX6p6quA3JJnXvxo49Zt8kH9L6DB3zW41aNszcwVnmgko5ERbD2sPLb7358BWIW79dpHACgmlkgnY0gmlwhKwSTDWKbXVsdGlhZGRyc5YACASsEkw1BuRQAAoErBJMNQbkUd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDbLggOHfaaBE0uxNhv9a-CExDkCTRt_vYribjqiLTfISDdGNwguRQg3VkcILkUoV3YWt1MgE"}'
2026-03-27 04:49:41.594 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:49:41.594 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:49:41.599 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:49:41.600 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:21222/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:49:41.608 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:49:41.609 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:8460/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:49:41.617 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:49:41.618 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:58447/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:49:41.625 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:49:41.627 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2104/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:41.634 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.140.157/tcp/45564/p2p/16Uiu2HAkzZ4AAgCw8NvKavsiSXQr18gKaPVfQTS3JAb42J6ahrDX","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:49:41.634 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:21222/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:41.637 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.69.208/tcp/2105/p2p/16Uiu2HAmVQrzurz6mVvbmBJV2ZnQajBV3fu9HWUAoekxoNrsDBUU","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.255.210/tcp/37032/p2p/16Uiu2HAmHDZiG9cxy3MB7kdy7zVdrqfy9HoCPrPuLBj1CBo1CJ9o","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:49:41.637 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:8460/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:41.639 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.76.53/tcp/34582/p2p/16Uiu2HAmKyPDwDucYdkzSBsMXL7ZBXCn7kHiwwRyx94NuYuoev1h","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.140.157/tcp/21223/p2p/16Uiu2HAkzZ4AAgCw8NvKavsiSXQr18gKaPVfQTS3JAb42J6ahrDX","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-27 04:49:41.640 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58447/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:49:41.643 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.255.210/tcp/8461/p2p/16Uiu2HAmHDZiG9cxy3MB7kdy7zVdrqfy9HoCPrPuLBj1CBo1CJ9o","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-27 04:49:41.643 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-27 04:50:01.645 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10099', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:50:01.658 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10099', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%']
2026-03-27 04:50:01.670 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10099', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-27 04:50:01.681 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8004: root refcnt 5 limit 1000 loss 50%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-27 04:50:01.681 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58447/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:50:01.684 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-27 04:50:01.684 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:01.695 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:01.696 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:09.103 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:09.104 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:12.240 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:12.240 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:12.455 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:12.456 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:13.525 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:13.529 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-27 04:50:43.533 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58447/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:50:43.536 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587001684690649,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587001696597426,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587009104204924,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587012240485985,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587012456420069,"ephemeral":false,"proof":""}]'
2026-03-27 04:50:43.537 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10099', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:50:43.550 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10099', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:50:43.561 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10099', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%', '75.0%']
2026-03-27 04:50:43.571 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:58447/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:50:43.573 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-27 04:50:43.574 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:43.584 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:43.585 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:49.999 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:50.000 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:50:57.376 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:50:57.376 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2104/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:51:17.523 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-27 04:51:17.524 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-27 04:51:17.524 DEBUG [src.node.waku_node] Stopping container with id 4e93a6e6806e
2026-03-27 04:51:18.093 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:51:18.094 DEBUG [src.node.waku_node] Stopping container with id 3efc55b407f4
2026-03-27 04:51:18.677 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:51:18.679 DEBUG [src.node.waku_node] Stopping container with id 41af71e6376c
2026-03-27 04:51:19.169 ERROR [src.node.docker_mananger] Max retries reached for container 4e93a6e6806e. Exiting log stream.
2026-03-27 04:51:19.261 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:51:19.264 DEBUG [src.node.waku_node] Stopping container with id 16b09238887f
2026-03-27 04:51:19.491 ERROR [src.node.docker_mananger] Max retries reached for container 3efc55b407f4. Exiting log stream.
2026-03-27 04:51:19.809 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:51:19.811 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-27 04:51:19.956 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:51:20.034 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:51:20.111 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:51:20.187 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:51:20.189 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-27 04:51:20.191 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-27 04:51:20.193 DEBUG [src.libs.common] Attaching file ./log/docker/node4_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.201 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.208 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.214 DEBUG [src.libs.common] Attaching file ./log/docker/node3_2026-03-27_04-49-36__4905e1d1-5065-46c9-9220-a63be5ebf058__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.227 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-27 04:51:20.228 DEBUG [tests.conftest] Running test: test_relay_packet_loss_correlated_vs_uncorrelated with id: 2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601
2026-03-27 04:51:20.229 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-27 04:51:20.229 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-27 04:51:20.241 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:51:20.242 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.251 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:51:20.252 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.262 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:51:20.263 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.272 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-27 04:51:20.272 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.273 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:51:20.273 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:51:20.276 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:51:20.277 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.176.67
2026-03-27 04:51:20.277 DEBUG [src.node.docker_mananger] Generated ports ['41277', '41278', '41279', '41280', '41281']
2026-03-27 04:51:20.277 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:51:20.277 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:51:20.277 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:51:20.278 DEBUG [src.node.docker_mananger] docker run -i -t -p 41277:41277 -p 41278:41278 -p 41279:41279 -p 41280:41280 -p 41281:41281 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=41279 --rest-port=41277 --tcp-port=41278 --discv5-udp-port=41280 --rest-address=0.0.0.0 --nat=extip:172.18.176.67 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=487afa36c8eaef4fdcaf5c5d40648dca4cf1cebf9caaeb4d0cb0da0fe1f26ec9 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=41281 --metrics-logging=true --relay=true
2026-03-27 04:51:20.316 ERROR [src.node.docker_mananger] Max retries reached for container 41af71e6376c. Exiting log stream.
2026-03-27 04:51:20.523 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.176.67 waku 7eb1958cd151745dbcc7451889bf21c0cd9611428b36de0e88ffb8d69a2e938f
2026-03-27 04:51:20.559 DEBUG [src.node.docker_mananger] Container started with ID 7eb1958cd151. Setting up logs at ./log/docker/node1_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:20.560 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 41277
2026-03-27 04:51:20.560 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:51:21.327 ERROR [src.node.docker_mananger] Max retries reached for container 16b09238887f. Exiting log stream.
2026-03-27 04:51:21.561 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41277/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:21.564 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:51:21.565 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:51:21.565 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41277/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:21.567 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.176.67/tcp/41278/p2p/16Uiu2HAmVRYVdNn5BRZhPmaGjzwN7Tggtcs1XFkcZEbLaHxFfD8o","/ip4/172.18.176.67/tcp/41279/ws/p2p/16Uiu2HAmVRYVdNn5BRZhPmaGjzwN7Tggtcs1XFkcZEbLaHxFfD8o"],"enrUri":"enr:-L24QOfupBUZfToSxX9Sljyl5QUieOiuyW68hN_GXf0GsVPSEXMTuk2pJn-GoxRV2XvdR9oG6omM6E19w-fjjNKCymUCgmlkgnY0gmlwhKwSsEOKbXVsdGlhZGRyc5YACASsErBDBqE-AAoErBKwQwahP90DgnJzhQADAQAAiXNlY3AyNTZrMaED-STkOHj0Uz2IG9sGP4a-DsAjiv7mcSIWNSU97tkOJySDdGNwgqE-g3VkcIKhQIV3YWt1MgE"}'
2026-03-27 04:51:21.568 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:51:21.568 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:51:21.568 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:51:21.569 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:51:21.569 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.125.245
2026-03-27 04:51:21.570 DEBUG [src.node.docker_mananger] Generated ports ['19896', '19897', '19898', '19899', '19900']
2026-03-27 04:51:21.570 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:51:21.570 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:51:21.570 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:51:21.570 DEBUG [src.node.docker_mananger] docker run -i -t -p 19896:19896 -p 19897:19897 -p 19898:19898 -p 19899:19899 -p 19900:19900 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=19898 --rest-port=19896 --tcp-port=19897 --discv5-udp-port=19899 --rest-address=0.0.0.0 --nat=extip:172.18.125.245 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=13da59ba862dafdd0de41139afb672fe13d72c61dd7bc22d8fc60ffe7be9ee47 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=19900 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QOfupBUZfToSxX9Sljyl5QUieOiuyW68hN_GXf0GsVPSEXMTuk2pJn-GoxRV2XvdR9oG6omM6E19w-fjjNKCymUCgmlkgnY0gmlwhKwSsEOKbXVsdGlhZGRyc5YACASsErBDBqE-AAoErBKwQwahP90DgnJzhQADAQAAiXNlY3AyNTZrMaED-STkOHj0Uz2IG9sGP4a-DsAjiv7mcSIWNSU97tkOJySDdGNwgqE-g3VkcIKhQIV3YWt1MgE
2026-03-27 04:51:21.765 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.125.245 waku 172e2e4eff20bf05b97553ee2826a8b2e4541540f4fb9c5886f4d1de244e7362
2026-03-27 04:51:21.801 DEBUG [src.node.docker_mananger] Container started with ID 172e2e4eff20. Setting up logs at ./log/docker/node2_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:21.801 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 19896
2026-03-27 04:51:21.801 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:51:22.803 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19896/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:22.828 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:51:22.829 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:51:22.829 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19896/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:22.837 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.125.245/tcp/19897/p2p/16Uiu2HAm6BcHdXzCFTmrVCnvgCC9WgkFgeiBWMdLRzMNEhp58YCk","/ip4/172.18.125.245/tcp/19898/ws/p2p/16Uiu2HAm6BcHdXzCFTmrVCnvgCC9WgkFgeiBWMdLRzMNEhp58YCk"],"enrUri":"enr:-L24QOyPQZ_KlSgTf4CAySR4_kk-IZDyVV2EdA0SubH68nmRdY2c2gKSDf0mhXwKshsYK54lGh-5ylvNfHDk5my33sACgmlkgnY0gmlwhKwSffWKbXVsdGlhZGRyc5YACASsEn31Bk25AAoErBJ99QZNut0DgnJzhQADAQAAiXNlY3AyNTZrMaECn9Y_8cGS9BzvEGeBGrnr-K3EbLnPFgSznf3h1_xc0T2DdGNwgk25g3VkcIJNu4V3YWt1MgE"}'
2026-03-27 04:51:22.839 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:51:22.839 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:51:22.840 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:51:22.842 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:51:22.842 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.199.100
2026-03-27 04:51:22.842 DEBUG [src.node.docker_mananger] Generated ports ['7986', '7987', '7988', '7989', '7990']
2026-03-27 04:51:22.843 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:51:22.843 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:51:22.843 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:51:22.843 DEBUG [src.node.docker_mananger] docker run -i -t -p 7986:7986 -p 7987:7987 -p 7988:7988 -p 7989:7989 -p 7990:7990 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=7988 --rest-port=7986 --tcp-port=7987 --discv5-udp-port=7989 --rest-address=0.0.0.0 --nat=extip:172.18.199.100 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=44b4fa64ae27eb57f85dcd64bb7d98fecab8d03cfdec94eea7d559f8567c198b --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=7990 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QOyPQZ_KlSgTf4CAySR4_kk-IZDyVV2EdA0SubH68nmRdY2c2gKSDf0mhXwKshsYK54lGh-5ylvNfHDk5my33sACgmlkgnY0gmlwhKwSffWKbXVsdGlhZGRyc5YACASsEn31Bk25AAoErBJ99QZNut0DgnJzhQADAQAAiXNlY3AyNTZrMaECn9Y_8cGS9BzvEGeBGrnr-K3EbLnPFgSznf3h1_xc0T2DdGNwgk25g3VkcIJNu4V3YWt1MgE
2026-03-27 04:51:23.037 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.199.100 waku 906608381eba86c8036354a4146e9b03a818b42b27acf634ee887efd64433bde
2026-03-27 04:51:23.073 DEBUG [src.node.docker_mananger] Container started with ID 906608381eba. Setting up logs at ./log/docker/node3_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:23.074 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 7986
2026-03-27 04:51:23.075 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:51:24.077 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7986/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:24.093 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:51:24.095 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:51:24.097 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7986/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:24.102 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.199.100/tcp/7987/p2p/16Uiu2HAm9sZGPt1Mp7CvqzcRRmjgYgdApVww9HcZrmtn6vDEtbvt","/ip4/172.18.199.100/tcp/7988/ws/p2p/16Uiu2HAm9sZGPt1Mp7CvqzcRRmjgYgdApVww9HcZrmtn6vDEtbvt"],"enrUri":"enr:-L24QEsDfFA-srDO3zLngRxC9mqNDm-WhP2BtH2L48YwJQXLRYqFya1g9byG3rejq6v67169mbrT7iwgl4J9_NC9YBYCgmlkgnY0gmlwhKwSx2SKbXVsdGlhZGRyc5YACASsEsdkBh8zAAoErBLHZAYfNN0DgnJzhQADAQAAiXNlY3AyNTZrMaEC1qUgcVX30u_O7ILL5TdttXAz0LujnkeD3NOAPoAwgQWDdGNwgh8zg3VkcIIfNYV3YWt1MgE"}'
2026-03-27 04:51:24.102 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:51:24.103 DEBUG [src.node.waku_node] Starting Node...
2026-03-27 04:51:24.103 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-27 04:51:24.107 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-27 04:51:24.107 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.112.151
2026-03-27 04:51:24.108 DEBUG [src.node.docker_mananger] Generated ports ['46691', '46692', '46693', '46694', '46695']
2026-03-27 04:51:24.108 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-27 04:51:24.108 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-27 04:51:24.108 DEBUG [src.node.waku_node] Using volumes []
2026-03-27 04:51:24.109 DEBUG [src.node.docker_mananger] docker run -i -t -p 46691:46691 -p 46692:46692 -p 46693:46693 -p 46694:46694 -p 46695:46695 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=46693 --rest-port=46691 --tcp-port=46692 --discv5-udp-port=46694 --rest-address=0.0.0.0 --nat=extip:172.18.112.151 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=f96cc0af11ebbcccac8bfee9bae5678bef0cfc08eaeb979f7bf3c504ed5bc0a3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=46695 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QEsDfFA-srDO3zLngRxC9mqNDm-WhP2BtH2L48YwJQXLRYqFya1g9byG3rejq6v67169mbrT7iwgl4J9_NC9YBYCgmlkgnY0gmlwhKwSx2SKbXVsdGlhZGRyc5YACASsEsdkBh8zAAoErBLHZAYfNN0DgnJzhQADAQAAiXNlY3AyNTZrMaEC1qUgcVX30u_O7ILL5TdttXAz0LujnkeD3NOAPoAwgQWDdGNwgh8zg3VkcIIfNYV3YWt1MgE
2026-03-27 04:51:24.307 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.112.151 waku 9858888e0213b5df2e1402d25e5987181451f4de2b04b6d847cc7e05a9772194
2026-03-27 04:51:24.341 DEBUG [src.node.docker_mananger] Container started with ID 9858888e0213. Setting up logs at ./log/docker/node4_2026-03-27_04-51-20__81bf0968-0124-4e1b-8115-e6d22d705601__wakuorg_nwaku:latest.log
2026-03-27 04:51:24.341 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 46691
2026-03-27 04:51:24.342 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-27 04:51:25.344 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/health" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:25.363 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-27 04:51:25.364 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-27 04:51:25.364 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:25.369 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.112.151/tcp/46692/p2p/16Uiu2HAm2xzQZfNbkPBfS1iCYC6bG3xFWAf9VS7rbPisQE8vGx8T","/ip4/172.18.112.151/tcp/46693/ws/p2p/16Uiu2HAm2xzQZfNbkPBfS1iCYC6bG3xFWAf9VS7rbPisQE8vGx8T"],"enrUri":"enr:-L24QDSj-caazBgooxB9r1Ce0Re_7CeuEelOB-ofp79CRQ4wDJr3gYD8gKphIH8yi2te8cLKgiyQ54CFPMR4R_xNCtMCgmlkgnY0gmlwhKwScJeKbXVsdGlhZGRyc5YACASsEnCXBrZkAAoErBJwlwa2Zd0DgnJzhQADAQAAiXNlY3AyNTZrMaECcAemLTgI71HLoKIcLsvebLCYa85pLxU-x8UfKr4xfzSDdGNwgrZkg3VkcIK2ZoV3YWt1MgE"}'
2026-03-27 04:51:25.370 INFO [src.node.waku_node] REST service is ready !!
2026-03-27 04:51:25.372 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:51:25.376 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:25.377 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:19896/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:51:25.384 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:25.386 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:7986/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:51:25.394 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:25.394 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46691/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-27 04:51:25.403 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:25.404 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:41277/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:25.409 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.125.245/tcp/39038/p2p/16Uiu2HAm6BcHdXzCFTmrVCnvgCC9WgkFgeiBWMdLRzMNEhp58YCk","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:51:25.409 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:19896/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:25.412 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.176.67/tcp/41278/p2p/16Uiu2HAmVRYVdNn5BRZhPmaGjzwN7Tggtcs1XFkcZEbLaHxFfD8o","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.199.100/tcp/49848/p2p/16Uiu2HAm9sZGPt1Mp7CvqzcRRmjgYgdApVww9HcZrmtn6vDEtbvt","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:51:25.412 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:7986/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:25.414 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.125.245/tcp/19897/p2p/16Uiu2HAm6BcHdXzCFTmrVCnvgCC9WgkFgeiBWMdLRzMNEhp58YCk","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"},{"multiaddr":"/ip4/172.18.112.151/tcp/57204/p2p/16Uiu2HAm2xzQZfNbkPBfS1iCYC6bG3xFWAf9VS7rbPisQE8vGx8T","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"UnknownOrigin"}]'
2026-03-27 04:51:25.415 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:25.417 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.199.100/tcp/7987/p2p/16Uiu2HAm9sZGPt1Mp7CvqzcRRmjgYgdApVww9HcZrmtn6vDEtbvt","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.38.0-rc.0-12-g0b8609","origin":"Discv5"}]'
2026-03-27 04:51:25.417 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-27 04:51:45.419 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:51:45.431 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%']
2026-03-27 04:51:45.443 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-27 04:51:45.453 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8006: root refcnt 5 limit 1000 loss 50%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-27 04:51:45.453 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:51:45.455 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-27 04:51:45.456 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:51:46.928 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:46.928 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:51:47.993 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:47.995 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:51:49.224 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:49.224 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:51:49.440 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:49.440 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:51:50.896 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:51:50.896 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-27 04:52:20.896 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:52:20.899 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587105456250341,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587106928552661,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587107995484617,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587109224491100,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587109440501652,"ephemeral":false,"proof":""}]'
2026-03-27 04:52:20.901 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:52:20.913 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:52:20.925 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%', '75.0%']
2026-03-27 04:52:20.935 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:52:20.937 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-27 04:52:20.938 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:52:28.122 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:52:28.124 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:52:29.141 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:52:29.147 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:52:29.165 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:52:29.167 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:52:29.800 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:52:29.800 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:41277/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-27 04:52:29.809 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-27 04:52:29.811 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-27 04:52:59.811 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46691/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-27 04:52:59.814 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587140938112705,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587148124284737,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587149147376414,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587149167092168,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1774587149800495131,"ephemeral":false,"proof":""}]'
2026-03-27 04:52:59.816 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11930', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-27 04:52:59.826 DEBUG [tests.e2e.test_network_conditions] uncorrelated=5 correlated=5
2026-03-27 04:52:59.827 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-27 04:52:59.828 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-27 04:52:59.828 DEBUG [src.node.waku_node] Stopping container with id 7eb1958cd151
2026-03-27 04:53:00.429 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:53:00.431 DEBUG [src.node.waku_node] Stopping container with id 172e2e4eff20
2026-03-27 04:53:01.007 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:53:01.009 DEBUG [src.node.waku_node] Stopping container with id 906608381eba
2026-03-27 04:53:01.249 ERROR [src.node.docker_mananger] Max retries reached for container 7eb1958cd151. Exiting log stream.
2026-03-27 04:53:01.578 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:53:01.578 DEBUG [src.node.waku_node] Stopping container with id 9858888e0213
2026-03-27 04:53:02.115 ERROR [src.node.docker_mananger] Max retries reached for container 172e2e4eff20. Exiting log stream.
2026-03-27 04:53:02.131 DEBUG [src.node.waku_node] Container stopped.
2026-03-27 04:53:02.132 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-27 04:53:02.203 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:53:02.363 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:53:02.439 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-27 04:53:02.514 DEBUG [src.node.docker_mananger] No errors found in the waku logs.