490 lines
97 KiB
Plaintext

2026-03-13 04:35:28.697 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:35:28.697 DEBUG [tests.conftest] Running test: test_relay_packet_loss_correlated_vs_uncorrelated with id: 2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b
2026-03-13 04:35:28.697 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:35:28.698 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:35:28.705 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:35:28.705 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:28.711 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:35:28.711 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:28.717 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:35:28.718 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:28.724 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:35:28.724 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:28.724 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:35:28.724 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:35:28.726 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:35:28.726 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.81.101
2026-03-13 04:35:28.726 DEBUG [src.node.docker_mananger] Generated ports ['39951', '39952', '39953', '39954', '39955']
2026-03-13 04:35:28.726 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:35:28.726 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:35:28.726 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:35:28.727 DEBUG [src.node.docker_mananger] docker run -i -t -p 39951:39951 -p 39952:39952 -p 39953:39953 -p 39954:39954 -p 39955:39955 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=39953 --rest-port=39951 --tcp-port=39952 --discv5-udp-port=39954 --rest-address=0.0.0.0 --nat=extip:172.18.81.101 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=3446bdcba413daebad23a0207e23c683a0efe255fbdb9aefe65c27fd53fcd975 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=39955 --metrics-logging=true --relay=true
2026-03-13 04:35:28.897 ERROR [src.node.docker_mananger] Max retries reached for container 3e228b88e30b. Exiting log stream.
2026-03-13 04:35:28.914 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.81.101 waku daed7fc54b049d1572865aea6712a80ce946840c8ecb324966587734527fa3a6
2026-03-13 04:35:28.954 DEBUG [src.node.docker_mananger] Container started with ID daed7fc54b04. Setting up logs at ./log/docker/node1_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:28.954 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 39951
2026-03-13 04:35:28.957 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:35:29.492 ERROR [src.node.docker_mananger] Max retries reached for container ca294b4f0a56. Exiting log stream.
2026-03-13 04:35:29.957 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39951/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:29.961 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:35:29.961 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:35:29.961 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39951/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:29.964 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.81.101/tcp/39952/p2p/16Uiu2HAmNniebrkPhvaRfcaoyKnBM86b8dez1YAkvEGbbntzYupW","/ip4/172.18.81.101/tcp/39953/ws/p2p/16Uiu2HAmNniebrkPhvaRfcaoyKnBM86b8dez1YAkvEGbbntzYupW"],"enrUri":"enr:-L24QKMLysOV0in-s5I1JrFyIDT_Wp5NP3qtxi7KKz6wuXVIZpSPC7OT_o5UzT6-zJlvnlXiP8gfU2edZF2FDR_ArUACgmlkgnY0gmlwhKwSUWWKbXVsdGlhZGRyc5YACASsElFlBpwQAAoErBJRZQacEd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDlo_DEKT83FZPF8rQS0B4nhorgcRPNhDwZ22m4K1iypuDdGNwgpwQg3VkcIKcEoV3YWt1MgE"}'
2026-03-13 04:35:29.964 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:35:29.964 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:35:29.964 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:35:29.966 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:35:29.966 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.173.87
2026-03-13 04:35:29.966 DEBUG [src.node.docker_mananger] Generated ports ['43058', '43059', '43060', '43061', '43062']
2026-03-13 04:35:29.966 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:35:29.966 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:35:29.966 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:35:29.966 DEBUG [src.node.docker_mananger] docker run -i -t -p 43058:43058 -p 43059:43059 -p 43060:43060 -p 43061:43061 -p 43062:43062 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=43060 --rest-port=43058 --tcp-port=43059 --discv5-udp-port=43061 --rest-address=0.0.0.0 --nat=extip:172.18.173.87 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=8ffa3a5cc2eb14aac6fb9a1e3abdb07cc8c0caf3694ab7206df44529ebeb0ad2 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=43062 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QKMLysOV0in-s5I1JrFyIDT_Wp5NP3qtxi7KKz6wuXVIZpSPC7OT_o5UzT6-zJlvnlXiP8gfU2edZF2FDR_ArUACgmlkgnY0gmlwhKwSUWWKbXVsdGlhZGRyc5YACASsElFlBpwQAAoErBJRZQacEd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDlo_DEKT83FZPF8rQS0B4nhorgcRPNhDwZ22m4K1iypuDdGNwgpwQg3VkcIKcEoV3YWt1MgE
2026-03-13 04:35:30.163 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.173.87 waku a2c3ba893b102d6bdb275e6b448a2eb5366bb0ed0ef9624f473a4333408c2cb2
2026-03-13 04:35:30.202 DEBUG [src.node.docker_mananger] Container started with ID a2c3ba893b10. Setting up logs at ./log/docker/node2_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:30.202 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 43058
2026-03-13 04:35:30.203 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:35:31.203 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:43058/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:31.225 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:35:31.226 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:35:31.226 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:43058/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:31.234 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.173.87/tcp/43059/p2p/16Uiu2HAmKKuP1AoCz7fyUS4avYPgVg7HVsPVnqmkQaT5KWGZiRwP","/ip4/172.18.173.87/tcp/43060/ws/p2p/16Uiu2HAmKKuP1AoCz7fyUS4avYPgVg7HVsPVnqmkQaT5KWGZiRwP"],"enrUri":"enr:-L24QPrgnp7ek1Q8T0wlEOt5yNOwRj0TcXvBIxUm7dWuinPxDV3aDmMul-22BF4okLfqku4fYoUM1xwP2Sjpslc9x-kCgmlkgnY0gmlwhKwSrVeKbXVsdGlhZGRyc5YACASsEq1XBqgzAAoErBKtVwaoNN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDYx4nK1xgJl28WlUSmzOSHnnPHW8FqgPJHS8KNj0yxLqDdGNwgqgzg3VkcIKoNYV3YWt1MgE"}'
2026-03-13 04:35:31.234 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:35:31.235 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:35:31.236 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:35:31.239 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:35:31.239 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.78.110
2026-03-13 04:35:31.239 DEBUG [src.node.docker_mananger] Generated ports ['27381', '27382', '27383', '27384', '27385']
2026-03-13 04:35:31.239 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:35:31.239 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:35:31.239 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:35:31.239 DEBUG [src.node.docker_mananger] docker run -i -t -p 27381:27381 -p 27382:27382 -p 27383:27383 -p 27384:27384 -p 27385:27385 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=27383 --rest-port=27381 --tcp-port=27382 --discv5-udp-port=27384 --rest-address=0.0.0.0 --nat=extip:172.18.78.110 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=c33d199ab66cf43b299eb55ef0ce27d6fbae9ec234aa0cbfefa37cb1cf1cc1a0 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=27385 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QPrgnp7ek1Q8T0wlEOt5yNOwRj0TcXvBIxUm7dWuinPxDV3aDmMul-22BF4okLfqku4fYoUM1xwP2Sjpslc9x-kCgmlkgnY0gmlwhKwSrVeKbXVsdGlhZGRyc5YACASsEq1XBqgzAAoErBKtVwaoNN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDYx4nK1xgJl28WlUSmzOSHnnPHW8FqgPJHS8KNj0yxLqDdGNwgqgzg3VkcIKoNYV3YWt1MgE
2026-03-13 04:35:31.432 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.78.110 waku 20f1ffb6069bf35dd52b757cca8da2295bcb2643ee21d0ec05b1370f20f1a827
2026-03-13 04:35:31.468 DEBUG [src.node.docker_mananger] Container started with ID 20f1ffb6069b. Setting up logs at ./log/docker/node3_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:31.468 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 27381
2026-03-13 04:35:31.468 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:35:32.469 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27381/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:32.484 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:35:32.485 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:35:32.485 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27381/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:32.491 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.78.110/tcp/27382/p2p/16Uiu2HAmGNdxfaebs6eaiDHFULWnbxkzdRWTWMUyTrsQQXJhd2QP","/ip4/172.18.78.110/tcp/27383/ws/p2p/16Uiu2HAmGNdxfaebs6eaiDHFULWnbxkzdRWTWMUyTrsQQXJhd2QP"],"enrUri":"enr:-L24QN6JNj2zgq2YYxMIrvXb6j7TK8XEsNVEMXflBWixbNigM2tsq3DK7n8Af2jAUFr6YJsZCiQhe8P-HbrS_a4yL9oCgmlkgnY0gmlwhKwSTm6KbXVsdGlhZGRyc5YACASsEk5uBmr2AAoErBJObgZq990DgnJzhQADAQAAiXNlY3AyNTZrMaEDNz5mTqyhhMz0ifqAHiiZzfcvnw_e173zT5fLiPr6RjCDdGNwgmr2g3VkcIJq-IV3YWt1MgE"}'
2026-03-13 04:35:32.492 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:35:32.492 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:35:32.493 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:35:32.495 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:35:32.495 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.195.122
2026-03-13 04:35:32.495 DEBUG [src.node.docker_mananger] Generated ports ['49206', '49207', '49208', '49209', '49210']
2026-03-13 04:35:32.495 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:35:32.495 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:35:32.495 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:35:32.496 DEBUG [src.node.docker_mananger] docker run -i -t -p 49206:49206 -p 49207:49207 -p 49208:49208 -p 49209:49209 -p 49210:49210 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=49208 --rest-port=49206 --tcp-port=49207 --discv5-udp-port=49209 --rest-address=0.0.0.0 --nat=extip:172.18.195.122 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fb859d8a641fab9fcd8fa77fe1445907dd7aae20d8e9e35fe3e80f6619fbc509 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=49210 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QN6JNj2zgq2YYxMIrvXb6j7TK8XEsNVEMXflBWixbNigM2tsq3DK7n8Af2jAUFr6YJsZCiQhe8P-HbrS_a4yL9oCgmlkgnY0gmlwhKwSTm6KbXVsdGlhZGRyc5YACASsEk5uBmr2AAoErBJObgZq990DgnJzhQADAQAAiXNlY3AyNTZrMaEDNz5mTqyhhMz0ifqAHiiZzfcvnw_e173zT5fLiPr6RjCDdGNwgmr2g3VkcIJq-IV3YWt1MgE
2026-03-13 04:35:32.704 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.195.122 waku fd6eb7c9a12f2c5e6831c3882ca5dd5c371585c6a5d80835198e5c2296e27c8e
2026-03-13 04:35:32.738 DEBUG [src.node.docker_mananger] Container started with ID fd6eb7c9a12f. Setting up logs at ./log/docker/node4_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:35:32.739 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 49206
2026-03-13 04:35:32.740 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:35:33.741 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49206/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:33.761 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:35:33.762 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:35:33.762 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49206/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:33.769 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.195.122/tcp/49207/p2p/16Uiu2HAmHdb9hfXPx6s8AW5uZwY7Tki72K8EPBKnvjk8ybM6wXMo","/ip4/172.18.195.122/tcp/49208/ws/p2p/16Uiu2HAmHdb9hfXPx6s8AW5uZwY7Tki72K8EPBKnvjk8ybM6wXMo"],"enrUri":"enr:-L24QLonG8sHjZ1l_5QkgSrIYbkay6KVI6GEhQZ-TuGlGSJZdPDPq7Eo0jLp3jBa5NwPr43-MAhZ8x6IyRFnP4736noCgmlkgnY0gmlwhKwSw3qKbXVsdGlhZGRyc5YACASsEsN6BsA3AAoErBLDegbAON0DgnJzhQADAQAAiXNlY3AyNTZrMaEDSe6fKfRCt8-oA38CZEbhfw3M0VRr_2AAUgnYyt9O6q6DdGNwgsA3g3VkcILAOYV3YWt1MgE"}'
2026-03-13 04:35:33.771 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:35:33.772 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:35:33.777 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:33.778 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:43058/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:35:33.783 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:33.786 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:27381/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:35:33.794 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:33.795 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:49206/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:35:33.804 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:33.804 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39951/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:33.808 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.173.87/tcp/35084/p2p/16Uiu2HAmKKuP1AoCz7fyUS4avYPgVg7HVsPVnqmkQaT5KWGZiRwP","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-13 04:35:33.808 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:43058/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:33.812 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.78.110/tcp/39948/p2p/16Uiu2HAmGNdxfaebs6eaiDHFULWnbxkzdRWTWMUyTrsQQXJhd2QP","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.81.101/tcp/39952/p2p/16Uiu2HAmNniebrkPhvaRfcaoyKnBM86b8dez1YAkvEGbbntzYupW","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:35:33.812 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:27381/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:33.815 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.195.122/tcp/50042/p2p/16Uiu2HAmHdb9hfXPx6s8AW5uZwY7Tki72K8EPBKnvjk8ybM6wXMo","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.173.87/tcp/43059/p2p/16Uiu2HAmKKuP1AoCz7fyUS4avYPgVg7HVsPVnqmkQaT5KWGZiRwP","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:35:33.815 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49206/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:33.818 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.78.110/tcp/27382/p2p/16Uiu2HAmGNdxfaebs6eaiDHFULWnbxkzdRWTWMUyTrsQQXJhd2QP","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:35:33.818 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-13 04:35:53.820 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8225', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:35:53.832 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8225', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%']
2026-03-13 04:35:53.844 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8225', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-13 04:35:53.854 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8002: root refcnt 5 limit 1000 loss 50%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-13 04:35:53.854 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49206/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:35:53.857 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-13 04:35:53.857 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:35:53.866 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:53.871 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:35:53.884 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:53.885 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:35:53.899 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:53.901 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:35:55.946 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:55.951 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:35:55.968 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:35:55.971 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-13 04:36:25.971 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49206/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:36:25.974 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376553857815651,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376553871843448,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376553885555635,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376553900954599,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376555950882584,"ephemeral":false,"proof":""}]'
2026-03-13 04:36:25.976 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8225', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:36:25.987 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8225', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:36:25.997 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '8225', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%', '75.0%']
2026-03-13 04:36:26.007 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:49206/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:36:26.010 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-13 04:36:26.010 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:36:37.635 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:36:37.636 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:36:39.077 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:36:39.078 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:36:39.093 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:36:39.096 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39951/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:36:59.376 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:36:59.377 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:36:59.377 DEBUG [src.node.waku_node] Stopping container with id daed7fc54b04
2026-03-13 04:36:59.958 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:36:59.959 DEBUG [src.node.waku_node] Stopping container with id a2c3ba893b10
2026-03-13 04:37:00.534 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:37:00.536 DEBUG [src.node.waku_node] Stopping container with id 20f1ffb6069b
2026-03-13 04:37:00.775 ERROR [src.node.docker_mananger] Max retries reached for container daed7fc54b04. Exiting log stream.
2026-03-13 04:37:01.100 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:37:01.102 DEBUG [src.node.waku_node] Stopping container with id fd6eb7c9a12f
2026-03-13 04:37:01.351 ERROR [src.node.docker_mananger] Max retries reached for container a2c3ba893b10. Exiting log stream.
2026-03-13 04:37:01.660 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:37:01.663 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:37:01.732 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:37:01.799 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:37:01.866 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:37:01.938 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:37:01.939 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-13 04:37:01.940 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-13 04:37:01.941 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:37:01.945 DEBUG [src.libs.common] Attaching file ./log/docker/node3_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:37:01.950 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:37:01.953 DEBUG [src.libs.common] Attaching file ./log/docker/node4_2026-03-13_04-35-28__c8bee6a4-3339-471f-bb0d-ff0d1958db5b__wakuorg_nwaku:latest.log
2026-03-13 04:37:01.966 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:37:01.967 DEBUG [tests.conftest] Running test: test_relay_packet_loss_correlated_vs_uncorrelated with id: 2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb
2026-03-13 04:37:01.967 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:37:01.968 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:37:01.980 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:37:01.980 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:01.990 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:37:01.990 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:02.002 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:37:02.002 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:02.011 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:37:02.012 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:02.012 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:37:02.012 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:37:02.015 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:37:02.015 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.13.255
2026-03-13 04:37:02.015 DEBUG [src.node.docker_mananger] Generated ports ['46143', '46144', '46145', '46146', '46147']
2026-03-13 04:37:02.016 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:37:02.016 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:37:02.016 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:37:02.017 DEBUG [src.node.docker_mananger] docker run -i -t -p 46143:46143 -p 46144:46144 -p 46145:46145 -p 46146:46146 -p 46147:46147 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=46145 --rest-port=46143 --tcp-port=46144 --discv5-udp-port=46146 --rest-address=0.0.0.0 --nat=extip:172.18.13.255 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=837784bcc0f5adfebaa9a8dc1179ea31badaabc408df36cfb9edd13de4fad9d3 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=46147 --metrics-logging=true --relay=true
2026-03-13 04:37:02.071 ERROR [src.node.docker_mananger] Max retries reached for container 20f1ffb6069b. Exiting log stream.
2026-03-13 04:37:02.253 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.13.255 waku 56efcdaa784c8a5eea48ebb3242f26fe1398054a4048ef86802c28ca3b682470
2026-03-13 04:37:02.292 DEBUG [src.node.docker_mananger] Container started with ID 56efcdaa784c. Setting up logs at ./log/docker/node1_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:02.293 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 46143
2026-03-13 04:37:02.293 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:37:03.034 ERROR [src.node.docker_mananger] Max retries reached for container fd6eb7c9a12f. Exiting log stream.
2026-03-13 04:37:03.294 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46143/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:03.297 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:37:03.297 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:37:03.298 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46143/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:03.300 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.13.255/tcp/46144/p2p/16Uiu2HAm4aTpQf5uzfs7UbVdnPnyo1bmu86cG65azbcc61NJhcLU","/ip4/172.18.13.255/tcp/46145/ws/p2p/16Uiu2HAm4aTpQf5uzfs7UbVdnPnyo1bmu86cG65azbcc61NJhcLU"],"enrUri":"enr:-L24QLhCnqPVpFAX7ZYTQZLTIWR254S2UGrHiXoeAMbTzRBhT7D_7M_a7AwIE0dyq3eJyyjCGpYmPrmGUTbaKudvqnoCgmlkgnY0gmlwhKwSDf-KbXVsdGlhZGRyc5YACASsEg3_BrRAAAoErBIN_wa0Qd0DgnJzhQADAQAAiXNlY3AyNTZrMaECh_mogRE2aj460roNoLCYN7cUFkoPnMKjjIYx4UD00EWDdGNwgrRAg3VkcIK0QoV3YWt1MgE"}'
2026-03-13 04:37:03.300 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:37:03.300 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:37:03.301 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:37:03.302 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:37:03.302 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.76.52
2026-03-13 04:37:03.302 DEBUG [src.node.docker_mananger] Generated ports ['30435', '30436', '30437', '30438', '30439']
2026-03-13 04:37:03.302 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:37:03.303 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:37:03.303 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:37:03.303 DEBUG [src.node.docker_mananger] docker run -i -t -p 30435:30435 -p 30436:30436 -p 30437:30437 -p 30438:30438 -p 30439:30439 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=30437 --rest-port=30435 --tcp-port=30436 --discv5-udp-port=30438 --rest-address=0.0.0.0 --nat=extip:172.18.76.52 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=fccc7a8449e5bcaca8ecaf1e863a4bab4df27fa5e11304abe8ff4088bdc374cb --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=30439 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLhCnqPVpFAX7ZYTQZLTIWR254S2UGrHiXoeAMbTzRBhT7D_7M_a7AwIE0dyq3eJyyjCGpYmPrmGUTbaKudvqnoCgmlkgnY0gmlwhKwSDf-KbXVsdGlhZGRyc5YACASsEg3_BrRAAAoErBIN_wa0Qd0DgnJzhQADAQAAiXNlY3AyNTZrMaECh_mogRE2aj460roNoLCYN7cUFkoPnMKjjIYx4UD00EWDdGNwgrRAg3VkcIK0QoV3YWt1MgE
2026-03-13 04:37:03.496 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.76.52 waku 1587c2da4a6996f1e2847d217b9a8cdd363ddfd9d55bf1cfaae6e5168bfa4d75
2026-03-13 04:37:03.531 DEBUG [src.node.docker_mananger] Container started with ID 1587c2da4a69. Setting up logs at ./log/docker/node2_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:03.531 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 30435
2026-03-13 04:37:03.532 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:37:04.532 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30435/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:04.547 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:37:04.551 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:37:04.552 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30435/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:04.556 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.76.52/tcp/30436/p2p/16Uiu2HAky3yBpFhy9ABbB8M2ssjNa5cvvJHpbYEPRUbpJta17A9b","/ip4/172.18.76.52/tcp/30437/ws/p2p/16Uiu2HAky3yBpFhy9ABbB8M2ssjNa5cvvJHpbYEPRUbpJta17A9b"],"enrUri":"enr:-L24QGJ2NmMExUcC3j_CN2u7v6peyXq5RcZr4fq0KvIw6MnnM9p5AQpINdtqNIWtC_NB8wNT8PKd0KdpDWoD4f6Qv5wCgmlkgnY0gmlwhKwSTDSKbXVsdGlhZGRyc5YACASsEkw0BnbkAAoErBJMNAZ25d0DgnJzhQADAQAAiXNlY3AyNTZrMaECNd9tHbs095wqIwe0UgOFAaK1HssOY0RF7C4k75X1PwaDdGNwgnbkg3VkcIJ25oV3YWt1MgE"}'
2026-03-13 04:37:04.556 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:37:04.557 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:37:04.557 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:37:04.560 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:37:04.561 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.23.187
2026-03-13 04:37:04.561 DEBUG [src.node.docker_mananger] Generated ports ['44225', '44226', '44227', '44228', '44229']
2026-03-13 04:37:04.561 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:37:04.561 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:37:04.561 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:37:04.561 DEBUG [src.node.docker_mananger] docker run -i -t -p 44225:44225 -p 44226:44226 -p 44227:44227 -p 44228:44228 -p 44229:44229 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=44227 --rest-port=44225 --tcp-port=44226 --discv5-udp-port=44228 --rest-address=0.0.0.0 --nat=extip:172.18.23.187 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=a7a1f98adc6cdcfc6c8cd7247b70fcaa73d7c4bdaecbccfd1fd5a2dbd58d414d --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=44229 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QGJ2NmMExUcC3j_CN2u7v6peyXq5RcZr4fq0KvIw6MnnM9p5AQpINdtqNIWtC_NB8wNT8PKd0KdpDWoD4f6Qv5wCgmlkgnY0gmlwhKwSTDSKbXVsdGlhZGRyc5YACASsEkw0BnbkAAoErBJMNAZ25d0DgnJzhQADAQAAiXNlY3AyNTZrMaECNd9tHbs095wqIwe0UgOFAaK1HssOY0RF7C4k75X1PwaDdGNwgnbkg3VkcIJ25oV3YWt1MgE
2026-03-13 04:37:04.767 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.23.187 waku adf1fda89ba746ee0aa81c1471d663aaa831af9f2e1fac197b25339534fe99fc
2026-03-13 04:37:04.806 DEBUG [src.node.docker_mananger] Container started with ID adf1fda89ba7. Setting up logs at ./log/docker/node3_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:04.806 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 44225
2026-03-13 04:37:04.806 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:37:05.807 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44225/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:05.822 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:37:05.822 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:37:05.823 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44225/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:05.831 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.23.187/tcp/44226/p2p/16Uiu2HAmCPPZ9ZeKfLzVDzeVzsJH2m7AoVEqkfzy5iiAM6phHfxK","/ip4/172.18.23.187/tcp/44227/ws/p2p/16Uiu2HAmCPPZ9ZeKfLzVDzeVzsJH2m7AoVEqkfzy5iiAM6phHfxK"],"enrUri":"enr:-L24QL4_TgG6VGt5afwwN_ulLILzVgwyXbuYOtwxJx1gJZxEOwu1pQ51u68gC-78y7yqLP7Kxnn5eBa0lDIDwnMO-_MCgmlkgnY0gmlwhKwSF7uKbXVsdGlhZGRyc5YACASsEhe7BqzCAAoErBIXuwasw90DgnJzhQADAQAAiXNlY3AyNTZrMaEC_ADxxXb8s57iRCWU661lw83Iftd-3dO91ZLB1uozzYCDdGNwgqzCg3VkcIKsxIV3YWt1MgE"}'
2026-03-13 04:37:05.831 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:37:05.832 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:37:05.832 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:37:05.835 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:37:05.835 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.54.154
2026-03-13 04:37:05.835 DEBUG [src.node.docker_mananger] Generated ports ['2016', '2017', '2018', '2019', '2020']
2026-03-13 04:37:05.835 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:37:05.835 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:37:05.835 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:37:05.835 DEBUG [src.node.docker_mananger] docker run -i -t -p 2016:2016 -p 2017:2017 -p 2018:2018 -p 2019:2019 -p 2020:2020 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=2018 --rest-port=2016 --tcp-port=2017 --discv5-udp-port=2019 --rest-address=0.0.0.0 --nat=extip:172.18.54.154 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=cbe544fc1bd65a9843cdd2a8ebf709aa2dddd9695dcc9f43d5a5feac08e9be06 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=2020 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QL4_TgG6VGt5afwwN_ulLILzVgwyXbuYOtwxJx1gJZxEOwu1pQ51u68gC-78y7yqLP7Kxnn5eBa0lDIDwnMO-_MCgmlkgnY0gmlwhKwSF7uKbXVsdGlhZGRyc5YACASsEhe7BqzCAAoErBIXuwasw90DgnJzhQADAQAAiXNlY3AyNTZrMaEC_ADxxXb8s57iRCWU661lw83Iftd-3dO91ZLB1uozzYCDdGNwgqzCg3VkcIKsxIV3YWt1MgE
2026-03-13 04:37:06.042 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.54.154 waku 280cee2defb43cd9a0fc156a2ed136a08e6a532608dba31cb675a3483498bf2d
2026-03-13 04:37:06.077 DEBUG [src.node.docker_mananger] Container started with ID 280cee2defb4. Setting up logs at ./log/docker/node4_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:37:06.077 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 2016
2026-03-13 04:37:06.077 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:37:07.079 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2016/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:07.098 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:37:07.099 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:37:07.100 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2016/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:07.106 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.54.154/tcp/2017/p2p/16Uiu2HAmNtDsEZqNh4JAFRnxg3ypeu45wcgiCuMr5FBy6qoaxSib","/ip4/172.18.54.154/tcp/2018/ws/p2p/16Uiu2HAmNtDsEZqNh4JAFRnxg3ypeu45wcgiCuMr5FBy6qoaxSib"],"enrUri":"enr:-L24QIQyZ8gxHL1-yqXFwun3AyBopxXzLiiO7xQB9CgoFO8EGw9c0OQTiWLFC4jd2rbN5F80Mi-vVB0uLx2xBkKtjowCgmlkgnY0gmlwhKwSNpqKbXVsdGlhZGRyc5YACASsEjaaBgfhAAoErBI2mgYH4t0DgnJzhQADAQAAiXNlY3AyNTZrMaEDl_iz5CB6A7N5fNKuvmLx5e9Zv4RBAGIf4X9Xfr-PJtiDdGNwggfhg3VkcIIH44V3YWt1MgE"}'
2026-03-13 04:37:07.107 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:37:07.107 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:37:07.113 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:07.114 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:30435/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:37:07.122 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:07.126 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:44225/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:37:07.133 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:07.136 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:2016/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:37:07.142 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:07.143 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:46143/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:07.147 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.76.52/tcp/46050/p2p/16Uiu2HAky3yBpFhy9ABbB8M2ssjNa5cvvJHpbYEPRUbpJta17A9b","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-13 04:37:07.147 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:30435/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:07.150 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.23.187/tcp/58518/p2p/16Uiu2HAmCPPZ9ZeKfLzVDzeVzsJH2m7AoVEqkfzy5iiAM6phHfxK","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.13.255/tcp/46144/p2p/16Uiu2HAm4aTpQf5uzfs7UbVdnPnyo1bmu86cG65azbcc61NJhcLU","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:37:07.150 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:44225/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:07.153 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.76.52/tcp/30436/p2p/16Uiu2HAky3yBpFhy9ABbB8M2ssjNa5cvvJHpbYEPRUbpJta17A9b","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.18.54.154/tcp/33306/p2p/16Uiu2HAmNtDsEZqNh4JAFRnxg3ypeu45wcgiCuMr5FBy6qoaxSib","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-13 04:37:07.153 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2016/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:07.156 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.23.187/tcp/44226/p2p/16Uiu2HAmCPPZ9ZeKfLzVDzeVzsJH2m7AoVEqkfzy5iiAM6phHfxK","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:37:07.156 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-13 04:37:27.158 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10046', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:37:27.170 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10046', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%']
2026-03-13 04:37:27.181 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10046', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-13 04:37:27.191 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8004: root refcnt 5 limit 1000 loss 50%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-13 04:37:27.191 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2016/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:37:27.193 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-13 04:37:27.194 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:37:27.411 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:27.412 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:37:27.627 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:27.627 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:37:32.003 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:32.004 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:37:34.062 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:34.065 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:37:35.691 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:37:35.691 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-13 04:38:05.692 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2016/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:05.694 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376647194317380,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376647412008144,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376647627844772,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376652004104149,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376654065391316,"ephemeral":false,"proof":""}]'
2026-03-13 04:38:05.696 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10046', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:38:05.707 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10046', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:38:05.717 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '10046', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%', '75.0%']
2026-03-13 04:38:05.726 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:2016/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:05.729 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-13 04:38:05.729 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:46143/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:38:25.915 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:38:25.916 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:38:25.916 DEBUG [src.node.waku_node] Stopping container with id 56efcdaa784c
2026-03-13 04:38:26.504 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:38:26.507 DEBUG [src.node.waku_node] Stopping container with id 1587c2da4a69
2026-03-13 04:38:27.096 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:38:27.098 DEBUG [src.node.waku_node] Stopping container with id adf1fda89ba7
2026-03-13 04:38:27.324 ERROR [src.node.docker_mananger] Max retries reached for container 56efcdaa784c. Exiting log stream.
2026-03-13 04:38:27.660 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:38:27.661 DEBUG [src.node.waku_node] Stopping container with id 280cee2defb4
2026-03-13 04:38:27.911 ERROR [src.node.docker_mananger] Max retries reached for container 1587c2da4a69. Exiting log stream.
2026-03-13 04:38:28.200 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:38:28.202 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:38:28.268 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:38:28.331 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:38:28.454 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:38:28.514 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:38:28.516 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-13 04:38:28.516 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-13 04:38:28.517 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.519 DEBUG [src.libs.common] Attaching file ./log/docker/node4_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.520 DEBUG [src.libs.common] Attaching file ./log/docker/node3_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.523 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-13_04-37-01__c2d8b972-4d16-4dd4-9be8-ba96b7f4cceb__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.528 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-13 04:38:28.528 DEBUG [tests.conftest] Running test: test_relay_packet_loss_correlated_vs_uncorrelated with id: 2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7
2026-03-13 04:38:28.529 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-13 04:38:28.529 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-13 04:38:28.536 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:38:28.536 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.542 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:38:28.542 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.549 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:38:28.549 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.555 DEBUG [src.node.docker_mananger] Docker client initialized with image wakuorg/nwaku:latest
2026-03-13 04:38:28.555 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.555 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:38:28.555 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:38:28.556 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:38:28.557 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.114.10
2026-03-13 04:38:28.557 DEBUG [src.node.docker_mananger] Generated ports ['12737', '12738', '12739', '12740', '12741']
2026-03-13 04:38:28.557 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:38:28.557 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:38:28.557 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:38:28.557 DEBUG [src.node.docker_mananger] docker run -i -t -p 12737:12737 -p 12738:12738 -p 12739:12739 -p 12740:12740 -p 12741:12741 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=12739 --rest-port=12737 --tcp-port=12738 --discv5-udp-port=12740 --rest-address=0.0.0.0 --nat=extip:172.18.114.10 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=bcec4cb1eed7dec0e10fb666ad88d9cff4c844ffa2b561c1921cd21ecd18e671 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=12741 --metrics-logging=true --relay=true
2026-03-13 04:38:28.758 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.114.10 waku b5dd24df3b6ba4ce86e099888e2ee9028d2f0dd2e99b4d2586c478c43cc6e16f
2026-03-13 04:38:28.792 DEBUG [src.node.docker_mananger] Container started with ID b5dd24df3b6b. Setting up logs at ./log/docker/node1_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:28.793 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 12737
2026-03-13 04:38:28.793 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:38:28.799 ERROR [src.node.docker_mananger] Max retries reached for container adf1fda89ba7. Exiting log stream.
2026-03-13 04:38:29.124 ERROR [src.node.docker_mananger] Max retries reached for container 280cee2defb4. Exiting log stream.
2026-03-13 04:38:29.794 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12737/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:29.798 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:38:29.798 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:38:29.798 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12737/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:29.801 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.114.10/tcp/12738/p2p/16Uiu2HAm1SeztTmb3CYeWakrqnDxgVUoaLbLCEY5o6kbAuPaTp9j","/ip4/172.18.114.10/tcp/12739/ws/p2p/16Uiu2HAm1SeztTmb3CYeWakrqnDxgVUoaLbLCEY5o6kbAuPaTp9j"],"enrUri":"enr:-L24QKSGXMni0Xz729Mj6ISK5mVxa8YsjuQ9SdMkKm5Cs0G7cAY64igVYFqbfhFMuFoOlyY9DzwteBfETH3r5ihhCrECgmlkgnY0gmlwhKwScgqKbXVsdGlhZGRyc5YACASsEnIKBjHCAAoErBJyCgYxw90DgnJzhQADAQAAiXNlY3AyNTZrMaECWWaXzvt7iXzK1IFop8NU2eYu-sHD82F4ODVfkRrSJfaDdGNwgjHCg3VkcIIxxIV3YWt1MgE"}'
2026-03-13 04:38:29.801 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:38:29.801 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:38:29.801 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:38:29.803 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:38:29.803 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.207.20
2026-03-13 04:38:29.803 DEBUG [src.node.docker_mananger] Generated ports ['55089', '55090', '55091', '55092', '55093']
2026-03-13 04:38:29.803 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:38:29.803 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:38:29.803 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:38:29.804 DEBUG [src.node.docker_mananger] docker run -i -t -p 55089:55089 -p 55090:55090 -p 55091:55091 -p 55092:55092 -p 55093:55093 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=55091 --rest-port=55089 --tcp-port=55090 --discv5-udp-port=55092 --rest-address=0.0.0.0 --nat=extip:172.18.207.20 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=38657cefff3ba7ac9fffb6dcf70c2a5b687daaf8f95c3acb89ff9a7f692ee6cd --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=55093 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QKSGXMni0Xz729Mj6ISK5mVxa8YsjuQ9SdMkKm5Cs0G7cAY64igVYFqbfhFMuFoOlyY9DzwteBfETH3r5ihhCrECgmlkgnY0gmlwhKwScgqKbXVsdGlhZGRyc5YACASsEnIKBjHCAAoErBJyCgYxw90DgnJzhQADAQAAiXNlY3AyNTZrMaECWWaXzvt7iXzK1IFop8NU2eYu-sHD82F4ODVfkRrSJfaDdGNwgjHCg3VkcIIxxIV3YWt1MgE
2026-03-13 04:38:30.002 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.207.20 waku 9063d1869c90397a7cebe3d2d7df023aa738317fce1c6d63ce4b79b6006b6509
2026-03-13 04:38:30.038 DEBUG [src.node.docker_mananger] Container started with ID 9063d1869c90. Setting up logs at ./log/docker/node2_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:30.038 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 55089
2026-03-13 04:38:30.040 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:38:31.041 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55089/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:31.061 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:38:31.064 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:38:31.064 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55089/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:31.070 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.207.20/tcp/55090/p2p/16Uiu2HAmJoH4a5rVHwuVaQvz8gnq3uRzEJEv6nyVFRBCUrLZwaw5","/ip4/172.18.207.20/tcp/55091/ws/p2p/16Uiu2HAmJoH4a5rVHwuVaQvz8gnq3uRzEJEv6nyVFRBCUrLZwaw5"],"enrUri":"enr:-L24QLFwi0t4N6GWR2yU3r-kJtxZDBaKLbuQ7RTLz8V3qHXQSx5DB-wyXL92U6RJlkoz9Z4JunuhlJPcNKwq67js0awCgmlkgnY0gmlwhKwSzxSKbXVsdGlhZGRyc5YACASsEs8UBtcyAAoErBLPFAbXM90DgnJzhQADAQAAiXNlY3AyNTZrMaEDW0WqABSe9z6ZdISeYwUpQE0rdCTXZyBRiSYTaCTLK_SDdGNwgtcyg3VkcILXNIV3YWt1MgE"}'
2026-03-13 04:38:31.070 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:38:31.070 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:38:31.070 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:38:31.073 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:38:31.073 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.12.127
2026-03-13 04:38:31.073 DEBUG [src.node.docker_mananger] Generated ports ['5382', '5383', '5384', '5385', '5386']
2026-03-13 04:38:31.074 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:38:31.074 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:38:31.074 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:38:31.075 DEBUG [src.node.docker_mananger] docker run -i -t -p 5382:5382 -p 5383:5383 -p 5384:5384 -p 5385:5385 -p 5386:5386 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=5384 --rest-port=5382 --tcp-port=5383 --discv5-udp-port=5385 --rest-address=0.0.0.0 --nat=extip:172.18.12.127 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=c05c7a4dbeb7a301939706b10e9e69eef0bfcddbbba7d49bfebfd8e91a1befde --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=5386 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLFwi0t4N6GWR2yU3r-kJtxZDBaKLbuQ7RTLz8V3qHXQSx5DB-wyXL92U6RJlkoz9Z4JunuhlJPcNKwq67js0awCgmlkgnY0gmlwhKwSzxSKbXVsdGlhZGRyc5YACASsEs8UBtcyAAoErBLPFAbXM90DgnJzhQADAQAAiXNlY3AyNTZrMaEDW0WqABSe9z6ZdISeYwUpQE0rdCTXZyBRiSYTaCTLK_SDdGNwgtcyg3VkcILXNIV3YWt1MgE
2026-03-13 04:38:31.273 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.12.127 waku e98e52c000ccb882982fcd4c0e40bd1c2f77af64f678fa6817953a3699dd870b
2026-03-13 04:38:31.310 DEBUG [src.node.docker_mananger] Container started with ID e98e52c000cc. Setting up logs at ./log/docker/node3_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:31.310 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 5382
2026-03-13 04:38:31.311 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:38:32.311 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5382/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:32.329 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:38:32.332 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:38:32.333 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5382/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:32.336 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.12.127/tcp/5383/p2p/16Uiu2HAkwdk7yCXqXPJunkWX1qv9NPN6VLwJ1NcqcMeU8TosjgVL","/ip4/172.18.12.127/tcp/5384/ws/p2p/16Uiu2HAkwdk7yCXqXPJunkWX1qv9NPN6VLwJ1NcqcMeU8TosjgVL"],"enrUri":"enr:-L24QLzdktJ3FdFKnVjV--oazlhUc3NFeNhoahEMD2k10TgXOt1Dh0LPPvCpqKtzQ__Mvclie9Os2oRAvNJK1HnVI2ICgmlkgnY0gmlwhKwSDH-KbXVsdGlhZGRyc5YACASsEgx_BhUHAAoErBIMfwYVCN0DgnJzhQADAQAAiXNlY3AyNTZrMaECIM8GmO3QFcyb-Sc8SdSW1HIvJQQrOo16Q43lrvYREleDdGNwghUHg3VkcIIVCYV3YWt1MgE"}'
2026-03-13 04:38:32.337 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:38:32.338 DEBUG [src.node.waku_node] Starting Node...
2026-03-13 04:38:32.338 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-13 04:38:32.341 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-13 04:38:32.341 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.70.79
2026-03-13 04:38:32.341 DEBUG [src.node.docker_mananger] Generated ports ['26884', '26885', '26886', '26887', '26888']
2026-03-13 04:38:32.341 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-13 04:38:32.341 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-13 04:38:32.341 DEBUG [src.node.waku_node] Using volumes []
2026-03-13 04:38:32.341 DEBUG [src.node.docker_mananger] docker run -i -t -p 26884:26884 -p 26885:26885 -p 26886:26886 -p 26887:26887 -p 26888:26888 wakuorg/nwaku:latest --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=26886 --rest-port=26884 --tcp-port=26885 --discv5-udp-port=26887 --rest-address=0.0.0.0 --nat=extip:172.18.70.79 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=af082e1a542ebaeebe24bdeec493cccd9cda3dcabdbb815dfabe1b715f8da033 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=26888 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QLzdktJ3FdFKnVjV--oazlhUc3NFeNhoahEMD2k10TgXOt1Dh0LPPvCpqKtzQ__Mvclie9Os2oRAvNJK1HnVI2ICgmlkgnY0gmlwhKwSDH-KbXVsdGlhZGRyc5YACASsEgx_BhUHAAoErBIMfwYVCN0DgnJzhQADAQAAiXNlY3AyNTZrMaECIM8GmO3QFcyb-Sc8SdSW1HIvJQQrOo16Q43lrvYREleDdGNwghUHg3VkcIIVCYV3YWt1MgE
2026-03-13 04:38:32.546 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.70.79 waku a3c43ee10f711946d44b93b315e6a68855493d24c7192d78cf1969cc6931805a
2026-03-13 04:38:32.579 DEBUG [src.node.docker_mananger] Container started with ID a3c43ee10f71. Setting up logs at ./log/docker/node4_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:38:32.579 DEBUG [src.node.waku_node] Started container from image wakuorg/nwaku:latest. REST: 26884
2026-03-13 04:38:32.580 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-13 04:38:33.581 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26884/health" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:33.599 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-13 04:38:33.601 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-13 04:38:33.602 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26884/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:33.606 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.70.79/tcp/26885/p2p/16Uiu2HAmEizduNA6wpZyAtgD7EZZH2Em1iikSSmKsVYTHN9ZzhLc","/ip4/172.18.70.79/tcp/26886/ws/p2p/16Uiu2HAmEizduNA6wpZyAtgD7EZZH2Em1iikSSmKsVYTHN9ZzhLc"],"enrUri":"enr:-L24QAeeWHTUQS_XvzwBX3eSxlstNRVH4kAfi5O-wTX7dAGZcWHJ5f3vrGqm1ZWG4-XUwycK_lYMQSmMxXwxJ_fM2psCgmlkgnY0gmlwhKwSRk-KbXVsdGlhZGRyc5YACASsEkZPBmkFAAoErBJGTwZpBt0DgnJzhQADAQAAiXNlY3AyNTZrMaEDHr4FXnnb9HHoHcs0yk079WRXk6weVJLkl4eYuHncHzmDdGNwgmkFg3VkcIJpB4V3YWt1MgE"}'
2026-03-13 04:38:33.606 INFO [src.node.waku_node] REST service is ready !!
2026-03-13 04:38:33.607 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:38:33.611 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:33.613 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:55089/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:38:33.619 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:33.621 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:5382/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:38:33.628 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:33.629 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:26884/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-13 04:38:33.637 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:33.639 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:12737/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:33.645 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.207.20/tcp/46202/p2p/16Uiu2HAmJoH4a5rVHwuVaQvz8gnq3uRzEJEv6nyVFRBCUrLZwaw5","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-13 04:38:33.645 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:55089/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:33.647 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.114.10/tcp/12738/p2p/16Uiu2HAm1SeztTmb3CYeWakrqnDxgVUoaLbLCEY5o6kbAuPaTp9j","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"},{"multiaddr":"/ip4/172.18.12.127/tcp/45360/p2p/16Uiu2HAkwdk7yCXqXPJunkWX1qv9NPN6VLwJ1NcqcMeU8TosjgVL","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"}]'
2026-03-13 04:38:33.648 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:5382/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:33.650 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.70.79/tcp/33292/p2p/16Uiu2HAmEizduNA6wpZyAtgD7EZZH2Em1iikSSmKsVYTHN9ZzhLc","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.207.20/tcp/55090/p2p/16Uiu2HAmJoH4a5rVHwuVaQvz8gnq3uRzEJEv6nyVFRBCUrLZwaw5","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:38:33.650 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26884/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:33.653 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.12.127/tcp/5383/p2p/16Uiu2HAkwdk7yCXqXPJunkWX1qv9NPN6VLwJ1NcqcMeU8TosjgVL","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"logos-delivery-v0.36.0-164-g0ad551","origin":"Discv5"}]'
2026-03-13 04:38:33.653 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-13 04:38:53.655 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11870', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:38:53.668 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11870', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%']
2026-03-13 04:38:53.679 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11870', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-13 04:38:53.689 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8006: root refcnt 5 limit 1000 loss 50%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-13 04:38:53.689 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26884/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:38:53.692 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-13 04:38:53.692 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:38:53.907 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:53.908 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:38:54.961 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:54.961 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:38:56.187 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:56.187 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:38:57.667 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:38:57.668 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:39:00.387 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:39:00.388 DEBUG [src.libs.common] Sleeping for 30.0 seconds
2026-03-13 04:39:30.388 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26884/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:39:30.391 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376733692645358,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376733908122040,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376734961775695,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376736187765244,"ephemeral":false,"proof":""},{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1773376737668330027,"ephemeral":false,"proof":""}]'
2026-03-13 04:39:30.392 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11870', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:39:30.404 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11870', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-13 04:39:30.414 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '11870', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '50.0%', '75.0%']
2026-03-13 04:39:30.423 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:26884/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-13 04:39:30.425 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-13 04:39:30.426 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:39:34.571 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:39:34.577 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:39:37.851 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:39:37.852 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:39:37.861 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-13 04:39:37.862 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:12737/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-13 04:39:58.013 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-13 04:39:58.014 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-13 04:39:58.014 DEBUG [src.node.waku_node] Stopping container with id b5dd24df3b6b
2026-03-13 04:39:58.577 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:39:58.579 DEBUG [src.node.waku_node] Stopping container with id 9063d1869c90
2026-03-13 04:39:59.127 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:39:59.127 DEBUG [src.node.waku_node] Stopping container with id e98e52c000cc
2026-03-13 04:39:59.588 ERROR [src.node.docker_mananger] Max retries reached for container b5dd24df3b6b. Exiting log stream.
2026-03-13 04:39:59.667 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:39:59.667 DEBUG [src.node.waku_node] Stopping container with id a3c43ee10f71
2026-03-13 04:39:59.942 ERROR [src.node.docker_mananger] Max retries reached for container 9063d1869c90. Exiting log stream.
2026-03-13 04:40:00.244 DEBUG [src.node.waku_node] Container stopped.
2026-03-13 04:40:00.246 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-13 04:40:00.360 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:40:00.423 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:40:00.487 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:40:00.553 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-13 04:40:00.555 DEBUG [tests.conftest] Running fixture teardown: attach_logs_on_fail
2026-03-13 04:40:00.555 DEBUG [tests.conftest] Test failed, attempting to attach logs to the allure reports
2026-03-13 04:40:00.556 DEBUG [src.libs.common] Attaching file ./log/docker/node1_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:40:00.560 DEBUG [src.libs.common] Attaching file ./log/docker/node4_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:40:00.564 DEBUG [src.libs.common] Attaching file ./log/docker/node2_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log
2026-03-13 04:40:00.568 DEBUG [src.libs.common] Attaching file ./log/docker/node3_2026-03-13_04-38-28__c3781c45-17ac-48d9-9e2b-03a2c5bd93b7__wakuorg_nwaku:latest.log