2026-03-05 15:25:13 +00:00

148 lines
28 KiB
Plaintext

2026-03-05 15:04:42.681 DEBUG [tests.conftest] Running fixture setup: test_id
2026-03-05 15:04:42.682 DEBUG [tests.conftest] Running test: test_relay_4_nodes_sender_packet_loss with id: 2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87
2026-03-05 15:04:42.682 DEBUG [src.steps.common] Running fixture setup: common_setup
2026-03-05 15:04:42.682 DEBUG [src.steps.relay] Running fixture setup: relay_setup
2026-03-05 15:04:42.690 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:04:42.690 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node1_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:42.696 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:04:42.696 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node2_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:42.705 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:04:42.705 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node3_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:42.712 DEBUG [src.node.docker_mananger] Docker client initialized with image harbor.status.im/wakuorg/nwaku:v0.38.0-beta
2026-03-05 15:04:42.712 DEBUG [src.node.waku_node] WakuNode instance initialized with log path ./log/docker/node4_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:42.712 INFO [tests.e2e.test_network_conditions] Starting 4 nodes with relay enabled (bootstrap chain)
2026-03-05 15:04:42.712 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:04:42.712 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:04:42.714 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:04:42.714 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.6.90
2026-03-05 15:04:42.714 DEBUG [src.node.docker_mananger] Generated ports ['4572', '4573', '4574', '4575', '4576']
2026-03-05 15:04:42.715 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:04:42.715 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:04:42.715 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:04:42.715 DEBUG [src.node.docker_mananger] docker run -i -t -p 4572:4572 -p 4573:4573 -p 4574:4574 -p 4575:4575 -p 4576:4576 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4574 --rest-port=4572 --tcp-port=4573 --discv5-udp-port=4575 --rest-address=0.0.0.0 --nat=extip:172.18.6.90 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=64fac3de352a9aa8f78931c3946b8e5ef6cff4eb0beacd78cdf1ea2adc7e0a47 --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4576 --metrics-logging=true --relay=true
2026-03-05 15:04:42.904 ERROR [src.node.docker_mananger] Max retries reached for container 6563856171f5. Exiting log stream.
2026-03-05 15:04:42.915 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.6.90 waku 141a98d2f6c65c995024d972ff22ef52ff2c8a8b57b74389cda631ba3c41f1bd
2026-03-05 15:04:42.952 DEBUG [src.node.docker_mananger] Container started with ID 141a98d2f6c6. Setting up logs at ./log/docker/node1_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:42.953 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 4572
2026-03-05 15:04:42.955 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:04:43.482 ERROR [src.node.docker_mananger] Max retries reached for container be4d0ef6d0e0. Exiting log stream.
2026-03-05 15:04:43.956 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4572/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:43.960 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:04:43.960 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:04:43.960 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4572/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:43.962 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.6.90/tcp/4573/p2p/16Uiu2HAkvFU9Vh9Zznszn3w7yC7dHkb87LTJQdfjdDpZYHw3azuy","/ip4/172.18.6.90/tcp/4574/ws/p2p/16Uiu2HAkvFU9Vh9Zznszn3w7yC7dHkb87LTJQdfjdDpZYHw3azuy"],"enrUri":"enr:-L24QDbczU73iOHULQ1ByfYlBLrxabqDJgrYrCpJTOdAK1zbL5dVemjSWMnm9GOzX9g7Dsr1q30HJfw6sJyuWfC_09gCgmlkgnY0gmlwhKwSBlqKbXVsdGlhZGRyc5YACASsEgZaBhHdAAoErBIGWgYR3t0DgnJzhQADAQAAiXNlY3AyNTZrMaECDD5_4U3gtBxmakEuX5SgY9ZDLMTPiwk5-uuO5cwAoYyDdGNwghHdg3VkcIIR34V3YWt1MgE"}'
2026-03-05 15:04:43.963 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:04:43.963 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:04:43.963 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:04:43.964 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:04:43.965 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.45.19
2026-03-05 15:04:43.965 DEBUG [src.node.docker_mananger] Generated ports ['62543', '62544', '62545', '62546', '62547']
2026-03-05 15:04:43.965 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:04:43.965 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:04:43.965 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:04:43.965 DEBUG [src.node.docker_mananger] docker run -i -t -p 62543:62543 -p 62544:62544 -p 62545:62545 -p 62546:62546 -p 62547:62547 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=62545 --rest-port=62543 --tcp-port=62544 --discv5-udp-port=62546 --rest-address=0.0.0.0 --nat=extip:172.18.45.19 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=aea5cadacd2f9991dabfa3da31aef4daa3cbd11ba04c689a4c3ad5a0abafcffc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=62547 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QDbczU73iOHULQ1ByfYlBLrxabqDJgrYrCpJTOdAK1zbL5dVemjSWMnm9GOzX9g7Dsr1q30HJfw6sJyuWfC_09gCgmlkgnY0gmlwhKwSBlqKbXVsdGlhZGRyc5YACASsEgZaBhHdAAoErBIGWgYR3t0DgnJzhQADAQAAiXNlY3AyNTZrMaECDD5_4U3gtBxmakEuX5SgY9ZDLMTPiwk5-uuO5cwAoYyDdGNwghHdg3VkcIIR34V3YWt1MgE
2026-03-05 15:04:44.166 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.45.19 waku 7ab3a5d9d7064199795aecd9fc03e95dd3cc8bdc6742234127a93be5903187e2
2026-03-05 15:04:44.202 DEBUG [src.node.docker_mananger] Container started with ID 7ab3a5d9d706. Setting up logs at ./log/docker/node2_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:44.203 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 62543
2026-03-05 15:04:44.203 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:04:45.204 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62543/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:45.231 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:04:45.233 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:04:45.233 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62543/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:45.240 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.45.19/tcp/62544/p2p/16Uiu2HAmDvSbG1TPyGTJSSMUc9rv9FVh2yE5D2GHSFQTQHFs9UYT","/ip4/172.18.45.19/tcp/62545/ws/p2p/16Uiu2HAmDvSbG1TPyGTJSSMUc9rv9FVh2yE5D2GHSFQTQHFs9UYT"],"enrUri":"enr:-L24QIfuiLHR4rZr50Lx937xOAFFzN3Z5NP8scy2nN8HxBuKfnhqPRqSlvpjDpbMrt0UJyBd3bdjxdJjSpzJktDp6mICgmlkgnY0gmlwhKwSLROKbXVsdGlhZGRyc5YACASsEi0TBvRQAAoErBItEwb0Ud0DgnJzhQADAQAAiXNlY3AyNTZrMaEDEtER13V2iDayTkxV71Smv9GEnH7L2wJjfPDVu5XifIyDdGNwgvRQg3VkcIL0UoV3YWt1MgE"}'
2026-03-05 15:04:45.244 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:04:45.245 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:04:45.245 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:04:45.247 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:04:45.247 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.226.142
2026-03-05 15:04:45.247 DEBUG [src.node.docker_mananger] Generated ports ['39111', '39112', '39113', '39114', '39115']
2026-03-05 15:04:45.247 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:04:45.247 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:04:45.247 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:04:45.248 DEBUG [src.node.docker_mananger] docker run -i -t -p 39111:39111 -p 39112:39112 -p 39113:39113 -p 39114:39114 -p 39115:39115 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=39113 --rest-port=39111 --tcp-port=39112 --discv5-udp-port=39114 --rest-address=0.0.0.0 --nat=extip:172.18.226.142 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=e6c37c4f59149f79d22507ffe35cf117ac23bed3ebafe55ff5e31c310a175eed --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=39115 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QIfuiLHR4rZr50Lx937xOAFFzN3Z5NP8scy2nN8HxBuKfnhqPRqSlvpjDpbMrt0UJyBd3bdjxdJjSpzJktDp6mICgmlkgnY0gmlwhKwSLROKbXVsdGlhZGRyc5YACASsEi0TBvRQAAoErBItEwb0Ud0DgnJzhQADAQAAiXNlY3AyNTZrMaEDEtER13V2iDayTkxV71Smv9GEnH7L2wJjfPDVu5XifIyDdGNwgvRQg3VkcIL0UoV3YWt1MgE
2026-03-05 15:04:45.459 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.226.142 waku f593dbf96fdc1cf653f1b4fbff17f0e95ad4f7dcd61bf00765fb9580336285e2
2026-03-05 15:04:45.499 DEBUG [src.node.docker_mananger] Container started with ID f593dbf96fdc. Setting up logs at ./log/docker/node3_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:45.499 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 39111
2026-03-05 15:04:45.500 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:04:46.501 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39111/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:46.516 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:04:46.519 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:04:46.519 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39111/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:46.525 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.226.142/tcp/39112/p2p/16Uiu2HAmCoz6EEd37px5fQ2dhosBKdJWdgmHjxHnW7NLA3bioJUe","/ip4/172.18.226.142/tcp/39113/ws/p2p/16Uiu2HAmCoz6EEd37px5fQ2dhosBKdJWdgmHjxHnW7NLA3bioJUe"],"enrUri":"enr:-L24QGcUyJ2t0Ft9jF8PKqFpwL9o0w2AG-jUkwvH97ZIfNB1bOiQDfFXaE0XRX6s6yctjuGbr27o1CYWem-y4QBRdTICgmlkgnY0gmlwhKwS4o6KbXVsdGlhZGRyc5YACASsEuKOBpjIAAoErBLijgaYyd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDAk3vBRqIBsAnnsSQvBAN47difUhfui_RhMFnxc0i36eDdGNwgpjIg3VkcIKYyoV3YWt1MgE"}'
2026-03-05 15:04:46.525 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:04:46.525 DEBUG [src.node.waku_node] Starting Node...
2026-03-05 15:04:46.526 DEBUG [src.node.docker_mananger] Attempting to create or retrieve network waku
2026-03-05 15:04:46.529 DEBUG [src.node.docker_mananger] Network waku already exists
2026-03-05 15:04:46.529 DEBUG [src.node.docker_mananger] Generated random external IP 172.18.24.77
2026-03-05 15:04:46.529 DEBUG [src.node.docker_mananger] Generated ports ['4250', '4251', '4252', '4253', '4254']
2026-03-05 15:04:46.529 DEBUG [src.node.waku_node] RLN credentials were not set
2026-03-05 15:04:46.530 INFO [src.node.waku_node] RLN credentials not set or credential store not available, starting without RLN
2026-03-05 15:04:46.530 DEBUG [src.node.waku_node] Using volumes []
2026-03-05 15:04:46.530 DEBUG [src.node.docker_mananger] docker run -i -t -p 4250:4250 -p 4251:4251 -p 4252:4252 -p 4253:4253 -p 4254:4254 harbor.status.im/wakuorg/nwaku:v0.38.0-beta --listen-address=0.0.0.0 --rest=true --rest-admin=true --websocket-support=true --log-level=TRACE --rest-relay-cache-capacity=100 --websocket-port=4252 --rest-port=4250 --tcp-port=4251 --discv5-udp-port=4253 --rest-address=0.0.0.0 --nat=extip:172.18.24.77 --peer-exchange=true --discv5-discovery=true --cluster-id=3 --nodekey=9bc76beba62ddf394e7f6afdff98facf2aeb173d49eedafc5cfce9cc1e2d60bc --shard=0 --metrics-server=true --metrics-server-address=0.0.0.0 --metrics-server-port=4254 --metrics-logging=true --relay=true --discv5-bootstrap-node=enr:-L24QGcUyJ2t0Ft9jF8PKqFpwL9o0w2AG-jUkwvH97ZIfNB1bOiQDfFXaE0XRX6s6yctjuGbr27o1CYWem-y4QBRdTICgmlkgnY0gmlwhKwS4o6KbXVsdGlhZGRyc5YACASsEuKOBpjIAAoErBLijgaYyd0DgnJzhQADAQAAiXNlY3AyNTZrMaEDAk3vBRqIBsAnnsSQvBAN47difUhfui_RhMFnxc0i36eDdGNwgpjIg3VkcIKYyoV3YWt1MgE
2026-03-05 15:04:46.723 DEBUG [src.node.docker_mananger] docker network connect --ip 172.18.24.77 waku d4602767f01441d93ef29be28485a0156fe963f0553dbf2851fcab22ef122809
2026-03-05 15:04:46.762 DEBUG [src.node.docker_mananger] Container started with ID d4602767f014. Setting up logs at ./log/docker/node4_2026-03-05_15-04-42__66aa6235-9f3a-46c4-b48e-2df0210dff87__harbor.status.im_wakuorg_nwaku:v0.38.0-beta.log
2026-03-05 15:04:46.763 DEBUG [src.node.waku_node] Started container from image harbor.status.im/wakuorg/nwaku:v0.38.0-beta. REST: 4250
2026-03-05 15:04:46.763 DEBUG [src.libs.common] Sleeping for 1 seconds
2026-03-05 15:04:47.764 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4250/health" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:47.774 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"nodeHealth":"READY","connectionStatus":"Disconnected","protocolsHealth":[{"Relay":"NOT_READY","desc":"No connected peers"},{"Lightpush":"NOT_MOUNTED"},{"Legacy Lightpush":"NOT_MOUNTED"},{"Filter":"NOT_MOUNTED"},{"Store":"NOT_MOUNTED"},{"Legacy Store":"NOT_MOUNTED"},{"Peer Exchange":"READY"},{"Rendezvous":"NOT_READY","desc":"No Rendezvous peers are available yet"},{"Mix":"NOT_MOUNTED"},{"Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Legacy Lightpush Client":"NOT_READY","desc":"No Lightpush service peer available yet"},{"Store Client":"NOT_READY","desc":"No Store service peer available yet, neither Store service set up for the node"},{"Legacy Store Client":"NOT_READY","desc":"No Legacy Store service peers are available yet, neither Store service set up for the node"},{"Filter Client":"NOT_READY","desc":"No Filter service peer available yet"},{"Rln Relay":"NOT_MOUNTED"}]}'
2026-03-05 15:04:47.776 INFO [src.node.waku_node] Node protocols are initialized !!
2026-03-05 15:04:47.776 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4250/debug/v1/info" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:47.782 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'{"listenAddresses":["/ip4/172.18.24.77/tcp/4251/p2p/16Uiu2HAmLCwq9YtAbVcaDGVEzvKRdt8JohoYmdp44UHdVzqMnApb","/ip4/172.18.24.77/tcp/4252/ws/p2p/16Uiu2HAmLCwq9YtAbVcaDGVEzvKRdt8JohoYmdp44UHdVzqMnApb"],"enrUri":"enr:-L24QBKIa9SsBH_suSupKj9zyWXNHv6IZ7lSmnOQ-m8O-FvzFbyyqK5zFRWVb16OH1rDGKwYUgsk62V006pnxvYtf48CgmlkgnY0gmlwhKwSGE2KbXVsdGlhZGRyc5YACASsEhhNBhCbAAoErBIYTQYQnN0DgnJzhQADAQAAiXNlY3AyNTZrMaEDcDGLnK4CEieskMdkFdgbwBQ92YUDZ6sjlBwxfPBIz5SDdGNwghCbg3VkcIIQnYV3YWt1MgE"}'
2026-03-05 15:04:47.783 INFO [src.node.waku_node] REST service is ready !!
2026-03-05 15:04:47.783 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4572/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:04:47.790 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:04:47.791 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:62543/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:04:47.801 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:04:47.802 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:39111/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:04:47.810 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:04:47.811 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4250/relay/v1/subscriptions" -H "Content-Type: application/json" -d '["/waku/2/rs/3/1"]'
2026-03-05 15:04:47.821 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:04:47.821 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4572/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:47.825 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.45.19/tcp/32930/p2p/16Uiu2HAmDvSbG1TPyGTJSSMUc9rv9FVh2yE5D2GHSFQTQHFs9UYT","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.38.0-beta","origin":"UnknownOrigin"}]'
2026-03-05 15:04:47.826 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:62543/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:47.829 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.6.90/tcp/4573/p2p/16Uiu2HAkvFU9Vh9Zznszn3w7yC7dHkb87LTJQdfjdDpZYHw3azuy","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.38.0-beta","origin":"Discv5"},{"multiaddr":"/ip4/172.18.226.142/tcp/34208/p2p/16Uiu2HAmCoz6EEd37px5fQ2dhosBKdJWdgmHjxHnW7NLA3bioJUe","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.38.0-beta","origin":"UnknownOrigin"}]'
2026-03-05 15:04:47.829 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:39111/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:47.832 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.24.77/tcp/36568/p2p/16Uiu2HAmLCwq9YtAbVcaDGVEzvKRdt8JohoYmdp44UHdVzqMnApb","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.38.0-beta","origin":"UnknownOrigin"},{"multiaddr":"/ip4/172.18.45.19/tcp/62544/p2p/16Uiu2HAmDvSbG1TPyGTJSSMUc9rv9FVh2yE5D2GHSFQTQHFs9UYT","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.38.0-beta","origin":"Discv5"}]'
2026-03-05 15:04:47.832 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4250/admin/v1/peers" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:04:47.834 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"multiaddr":"/ip4/172.18.226.142/tcp/39112/p2p/16Uiu2HAmCoz6EEd37px5fQ2dhosBKdJWdgmHjxHnW7NLA3bioJUe","protocols":["/ipfs/id/1.0.0","/libp2p/autonat/1.0.0","/libp2p/circuit/relay/0.2.0/hop","/vac/waku/metadata/1.0.0","/vac/waku/relay/2.0.0","/vac/waku/rendezvous/1.0.0","/ipfs/ping/1.0.0","/vac/waku/filter-push/2.0.0-beta1","/vac/waku/peer-exchange/2.0.0-alpha1"],"shards":[0],"connected":"Connected","agent":"nwaku-v0.38.0-beta","origin":"Discv5"}]'
2026-03-05 15:04:47.835 DEBUG [src.libs.common] Sleeping for 20 seconds
2026-03-05 15:05:07.835 INFO [tests.e2e.test_network_conditions] Applying 30.0% packet loss on sender node1
2026-03-05 15:05:07.837 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4758', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-05 15:05:07.849 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4758', '-n', 'tc', 'qdisc', 'add', 'dev', 'eth0', 'root', 'netem', 'loss', '30.0%']
2026-03-05 15:05:07.860 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4758', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-05 15:05:07.870 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8002: root refcnt 5 limit 1000 loss 30%
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-05 15:05:07.870 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4250/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:05:07.873 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[]'
2026-03-05 15:05:07.873 INFO [src.node.api_clients.base_client] curl -v -X POST "http://127.0.0.1:4572/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d '{"payload": "UmVsYXkgd29ya3MhIQ==", "contentTopic": "/test/1/waku-relay/proto", "timestamp": '$(date +%s%N)'}'
2026-03-05 15:05:09.096 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'OK'
2026-03-05 15:05:09.098 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4758', '-n', 'tc', '-s', 'qdisc', 'show', 'dev', 'eth0']
2026-03-05 15:05:09.109 DEBUG [src.steps.network_conditions] tc -s qdisc show dev eth0:
qdisc netem 8002: root refcnt 5 limit 1000 loss 30%
Sent 500 bytes 5 pkt (dropped 3, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
2026-03-05 15:05:09.109 INFO [src.node.api_clients.base_client] curl -v -X GET "http://127.0.0.1:4250/relay/v1/messages/%2Fwaku%2F2%2Frs%2F3%2F1" -H "Content-Type: application/json" -d 'None'
2026-03-05 15:05:09.113 INFO [src.node.api_clients.base_client] Response status code: 200. Response content: b'[{"payload":"UmVsYXkgd29ya3MhIQ==","contentTopic":"/test/1/waku-relay/proto","version":0,"timestamp":1772723107873796656,"ephemeral":false,"proof":""}]'
2026-03-05 15:05:09.113 INFO [tests.e2e.test_network_conditions] Node4 received messages from node1 after 0 trails
2026-03-05 15:05:09.115 INFO [src.steps.network_conditions] TC exec: ['sudo', '-n', 'nsenter', '-t', '4758', '-n', 'tc', 'qdisc', 'del', 'dev', 'eth0', 'root']
2026-03-05 15:05:09.128 DEBUG [tests.conftest] Running fixture teardown: test_setup
2026-03-05 15:05:09.129 DEBUG [tests.conftest] Running fixture teardown: close_open_nodes
2026-03-05 15:05:09.130 DEBUG [src.node.waku_node] Stopping container with id 141a98d2f6c6
2026-03-05 15:05:09.765 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:05:09.765 DEBUG [src.node.waku_node] Stopping container with id 7ab3a5d9d706
2026-03-05 15:05:10.302 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:05:10.303 DEBUG [src.node.waku_node] Stopping container with id f593dbf96fdc
2026-03-05 15:05:10.580 ERROR [src.node.docker_mananger] Max retries reached for container 141a98d2f6c6. Exiting log stream.
2026-03-05 15:05:10.865 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:05:10.865 DEBUG [src.node.waku_node] Stopping container with id d4602767f014
2026-03-05 15:05:11.117 ERROR [src.node.docker_mananger] Max retries reached for container 7ab3a5d9d706. Exiting log stream.
2026-03-05 15:05:11.420 DEBUG [src.node.waku_node] Container stopped.
2026-03-05 15:05:11.424 DEBUG [tests.conftest] Running fixture teardown: check_waku_log_errors
2026-03-05 15:05:11.451 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 15:05:11.474 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 15:05:11.498 DEBUG [src.node.docker_mananger] No errors found in the waku logs.
2026-03-05 15:05:11.524 DEBUG [src.node.docker_mananger] No errors found in the waku logs.